SuperFed: Weight Shared Federated Learning
Authors: Alind Khare
Animesh Agrawal
Alexey Tumanov
Venue: Submitted to ICLR 2023 (2023)
Keywords: Weight Shared
Federated Learning

Abstract: Federated Learning (FL) is a well-established technique for privacy preserving
distributed training. Much attention has been given to various aspects of FL training. A growing number of applications that consume FL-trained models, however, increasingly operate under dynamically and unpredictably variable conditions, rendering a single model insufficient. We argue for training a global “family of models” cost efficiently in a federated fashion. Training them independently for different tradeoff points incurs ≈ O(k) cost for any k architectures of interest, however.
Straightforward applications of FL techniques to recent weight-shared training
approaches is either infeasible or prohibitively expensive. We propose SuperFed — an architectural framework that incurs O(1) cost to co-train a large family ofmodels in a federated fashion by leveraging weight-shared learning. We achieve an order of magnitude cost savings on both communication and computation by proposing two novel training mechanisms: (a) distribution of weight-shared models to federated clients, (b) central aggregation of arbitrarily overlapping weight-shared model parameters. The combination of these mechanisms is shown to reach an order of magnitude (9.43x) reduction in computation and communication cost for training a 5*10^18-sized family of models, compared to independently training as few as k = 9 DNNs without any accuracy loss.

Under review as a conference paper at ICLR 2023
SUPERFED: WEIGHT SHARED FEDERATED LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Federated Learning (FL) is well-established technique for privacy preserving distributed training. Much attention has been given to various aspects of FL train- ing. A growing number of applications that consume FL-trained models, however, increasingly operate under dynamically and unpredictably variable conditions, ren- dering single model insufficient. We argue for training global "family of models" cost efficiently in federated fashion. Training them independently for different tradeoff points incurs O(k) cost for any A architectures of interest, however. Straightforward applications of FL techniques to recent weight-shared training approaches is either infeasible or prohibitively expensive. We propose SuperFed- an architectural framework that incurs (1)) cost to co-train a large family of models a federated fashion by leveraging weight-shared learning. We achieve order of magnitude cost savings both communication and computation by proposing two novel training mechanisms: (a) distribution of weight-shard models to federated clients, (b) central aggregation of arbitrarily overlapping weight-shared model parameters. The combination of these mechanisms is shown to reach an order of magnitude (9.43x) reduction in computation and communication cost for training 5 100-sized family of models, compared to independently training as few as k 9 DNNs without any accuracy loss.
INTRODUCTION
With the increase in the computational power of smartphones, the use of on-device inference in mobile applications is on the rise, ranging from image recognition (google vision; azure vision). virtual assistant (Alexa), voice recognition (google ASR) to recommendation systems (Bin et al. 2019). Indeed, on-device inference is pervasive, especially with recent advances in software (Chen et al. 2018 torch mobile), accelerators (samsung exynos, apple neural engine), and neural architecture optimizations (Howard et al., 2019) Sun et al., 2020; Wu et al. 2019a). The surge in its use cases (Cai et al. 2017; Han et al., 2019 Kang et al. 2017 Lane et al. 2016, Reddi et al. 2021 Wu et al.) 2019b) has led to a growing interest in providing support not only for on-device inference, but also for on-device training of these models (Dhar et al. 2021).
Federated Learning (FL) is an emerging distributed training technique that allows smartphones with different data sources to collaboratively train ML model (McMahan et al., 2017, Chen & Chao 2020; Wang et al.) 2020; Karimireddy et al. 2021 Konečný et al. (016). FL. enjoys three key properties,
a) has smaller communication cost, b) is massively parallel, and c) involves no data-sharing. As a result, numerous applications such as GBoard (Hard et al., 2018), Apple's 's Siri (sir) 2019),, pharmaceutical discovery (CORDIS. 2019), medical imaging (Silva et al. 2019), health record mining (Huang & Liu 2019), and recommendation systems (Ammad-ud-dee et al. 2019) are readily adopting federated learning.
However, adoption of FL in smartphone applications is non-trivial. As result, recent works pay attention to the emerging challenges that occur in training, such as data heterogeneity (Karimireddy et al.) 2021; Li et al., 2020, Acarr et al. 2021), heterogeneous resources (Alistarh et al.) 2017; Ivkin et al. 2019; Li et al. 2020; Konečný et al. 2016), and privacy (Truex et al., 2019; Mo et al. 2021 Gong et al. 2021 ). These helped FL adoption, particularly in challenging training conditions. However, the success of FL adoption depends not only on tackling challenges that occur in training but also post-training (deployment). Indeed, deploying ML models for on-device inference is exceedingly challenging (Wu et al. 2019b, Reddi et al. 2021). Yet. most of the existing training techniques in FL do not take these deployment challenges into consideration. In this paper, we focus developing FL.

---

Under review as a conference paper at ICLR 2023
training algorithms specifically designed to address deployment challenges related to post-training inference.
It well-established that any single model statically chosen for on-device inference is sub-optimal. This is because the deployment conditions may continuously change on a multi-task system like smartphones (Xu et al. 2019) due to dynamic resource availability (Fang al. 2018). For instance, the computational budget may vary due to excessive consumption by background apps; the energy budget may vary if the smartphone is in low power or power-saver mode (Yu et al. 2019). Furthermore, an increasing number of applications require flexibility with respect to resource-accurast trade-off's in order efficiently utilize the dynamic resources in deployment (Fang et al., 2018). In all of these deployment scenarios, single model neither satisfies variable constraints nor offers the flexibility to make trade-offs.
In contrast to existing FL. approaches that produce a single model, we need to produce multiple model variants (varying in size/latency) for efficient on-device inference. However, training these model variants independently is computationally prohibitive (Cai al. 2020). This is particularly true for FL, where training these variants independently will cumulatively inflate the communication cost as well as the computation cost. Thus, it is imperative to develop techniques for training multiple models in a federated fastatoon cost efficiently without any accuracy loss achieving asymptotic cost improvements relative to independently training them.
To achieve this goal, we propose SuperFed- a novel fed- erated framework that targets the problem of efficient on- device inference on smartphones with better training algo- rithms. SuperFed co-trains family of model variants in
Federated Weight Shared Training
Deploy ment (Inference)
Local update
a federated fashion simultaneously by leveraging weight- sharing (Cai al.] (2020; Yu et al., 2020). After federated training. the clients perform local neural architecture search to find the appropriate model variants for their deployment scenarios. In weight-sharing. all model variants are sub- networks of a supernetwork (Cai et al., 2020) and share their parameters partially. The largest subnetwork's ss (or supernetwork's s) parameters contain other subnetworks' pa- rameters within it as proper subgraphs. There are two key benefits that weight-sharing brings in FL., it a) signif- icantly reduces the communication and computation cost for training k model variants, and b) requires no re- re-training after the federated training of the supernetwork is complete. Hence, SuperFed decouples training from neural architec- ture search which allows local clients to dynamically select subnetworks of their choice from the globally-traiee -trained su- pernetwork without any re-training.
Weight Staring Ft
Server
Weight shared FI. Training. Shared weights reside the server. NN subnetworks are distributed (left) and de- ployed (right) to participating clients, glob- ally training a dense accuracy-latence trade- off space.
Figure 1:
However, applying existing weight-shared training tech- niques to federated learning is challenging. First, weight- shared training techniques like Progressive Shrinking (PS) (Cai et al. 2020) work on centralized i.i.d data, whereas the data is decentralized and typically non-i.i.d in FL.. Second, PS uses a pre-trained largest subnetwork during the weight-shared training. This requirement becomes impractical in the context of FL as it a) hard obtain globally pre-trained FL model, or b) may significantly increase the overall communication cost to train it first. Third, weight-sharing training techniques need to minimize interference (Cai et al., 2020 Xu et al., 2022). Interference occurs when smaller subnetworks inter- fere with the training of larger subnetworks (Fig 2a in Yu et al. (2020)). To mitigate interference, PS adopts multi- thase training approach that prioritizes the training of larger subnetworks before training smaller subnetworks. Such multi-phase training may lead to significant communication cost in federated learning. Instead, we argue that the weight-shared training technique FI. must be be one-shot (single phase) to mitigate interference.
As a part of SuperFed framework, we propose MaxNet - a weight-shared training technique for FL. Figure provides high level overview of our proposed approach. MaxNet hosts the supernetwork in the server and assumes no prior pre-trained model before the federated training. MaxNet decides which individual subnetworks are distributed for training on participating clients and when (subnet- work distribution). MaxNet 's subnetwork distribution optimizes both lower bound (smallest subnet)
2

---

Under review as a conference paper at ICLR 2023
and upper bound (largest subnet) to increase the accuracy of every subnet in between. Since the subnetworks partially share their parameters, MaxNet also introduces a novel shared model weight aggregation mechanism that mitigates interference in weight-shared FL training. To summarize, contributions are as follows:
SuperFed: A weight shared training framework for FL that trains a family of model variants (DNN models), cost-efficiently in federated fashion.
MaxNet's 's subnetwork distribution: a heuristic that trains the upper and lower bounds in the model family by optimizing both bounds and load balancing their distribution over time. MaxNet's Aggregation: A mechanism for subnetwork weight aggregation with variable overlaps mitigating intra-network interference.
We perform rigorous evaluation of SuperFed 's weight-shared training technique MaxNet against non-weight shared baselines, where each model variant is trained independently in a federated fashion over CIFAR10/100 and CINIC-10 (downsampled ImageNet images) datasets. MaxNet trains a family of x 5 * 10th subnets showing 9.43x lower computational and 10.94x lower communication cost compared to training as few as 9 subnetworks separately with FedAvg. This order of magnitude reduction in training cost is achieved with no accuracy loss compared to independently trained subnetworks (without weight-sharing) with FedAvg.
1.1 RELATED WORK
In this section, we describe FL approaches that are closely related to SuperFed. We provide a background of weight shared training in non-federated environments in App. [ FL-training challenges. Our proposed work targets challenge of efficient deployment of FL-models for on-device inference which is fundamentally different from challenges that occur during training. Specifically, SuperFed aims to reduce the cost of training & global model variants in FL whereas existing works train a single model in FL-training. FedAvg (McMahan et al,,,017, 2017) and local SGD) (Lin et al. (020b) train single model by averaging model updates received from the participating clients. We show that FedAvg is a special case of SuperFed in App. C Recent modifications to FedAvg like FedDyn (Acar et al.) 2021). FedProx (Li et al.) 2020). Scaffold (Karimireddy et al., 2021) modify client local training to minimize communication cost for non-i.i.d client datasets. These techniques are complimentary to the proposed work and incorporating them in SuperFed framework is left as future work. Other techniques like Diao et al. (2020); Lin et al. (2020a) enable FL clients to train neural-nets (NNs) that differ in architecture enabling system heterogeneity in FL. SuperFed also allows clients to train architecturally different NNs during training. However, the goal of SuperFed is not enable system-hetert -eetr-eeeetrr but satisfy varying deployment scenarios of clients after training. Hence, SuperFed provides k global models to clients during on on-device inference, whereas, Diao et al. (2020); Lin al. (2020a) provide single global model every client during inference ppost-trainin!! Developing system-hhterogeeneity aware weight-shared FL training is future work.
AutoML in FL. AutoMI. for federated learning is an emerging field. SuperFed automates the neural architecture selection for varied client deployment scenarios. A known AutoMI. technique FedNAS (He et al. 2020) automatically searches for NN architecture that maximizes global accuracy in FL. We argue that the goal of SuperFed fundamentally differs from that of FedNAS. FedNAS doesn't 't target varying deployment scenarios of FL clients whereas SuperFed does. Hence, SuperFed trains k global model variants while FedNAS trains single global model. SuperFed decouples training from searching of NN architectures whereas FedNAS doesn't. This decoupling is important to allow clients run local NAS after training in SuperFed. Another line of work FedEx (Khodak al.) 202T) automates hyper-parameter search in FL. Similar to SuperFed, it uses weight-sharing to reduce the communication cost. However, FedEx uses weight-sharing in a different context than that of SuperFed. Specifically, FedEx shares parameter of an NN across different learning rate schedules, momentum and weight-decay. Whereas, SuperFed shares parameters of an NN across different NNs that differ in shape and sizes (similar to Cai et al. (2020). Moreover, FedEx produces a single global model at the end of hyper-parameter optimization in FL. Whereas, SuperFed produces k global models in FL.
like
Multi-Model FL. Another line of FL-research trains multiple models simultaneously but on a fairly different FL scenario than SuperFed. Multi-Modes FL (Bhuyan & Moharir 2022; da Silva et al. 2022) trains l global models (simultancously) that make predictions on different tasks (for e.g. one model may predict birds" species while another predicts dogs' species). Whereas, SuperFed trains k
3

---

Under review as a conference paper at ICLR 2023
Notation
Explanation
S
The set of clients in FL training, S .,, Client Participation ratio in FL.
C
h
Spatial heuristic for subnetwork distribution such that h S -
H(t)
Spatio-temporal heuristic for subnetwork distribution such that H: 21,
M(W arch .w) Weight subnetwork arch partially replacing weight of supernetwork
Table 1:
Notations used in Algorithms and 2
Full list of notations are described in Tab.
different global models a single task (every model in SuperFed makes prediction for the same classes) for varying client deployment scenarios. Multi-Model FL assumes availability of k different datasets client device whereas SuperFed trains every model on single (and same) dataset per client.
2 SUPERFED: WEIGHT SHARED FL FRAMEWORK
We start by describing our weight shared FL framework in this section. It provides pluggable algorithmic and implementation base for subnetwork distribution and aggregation heuristics between participating clients- novel challenge introduced by weight shared FL. We also propose one such SuperFed's 's heuristic MaxNet which instantiates a specific approach to subnetwork distribution and aggregation. Overall, the goal of weight-shared FL. training to train all subnetworks contained inside supernetwokk on all the data partitions. The problem formulation of weight-shared FL. is discussed in detail in App. Tab. |lists some notations used to describe SuperFed.
2.1 FEDERATED SETUP
Fundamentally, to train a supernetwork in a federated fashion, a server must decide how to distribute its sub-architectures between participating clients. This happens the beginning of each FL round (line 4 of algorithm M]. The server extracts the subnetworks from the hosted supernet and sends them the clients (algorithm line 6). Each participating client trains the received subnetwork for fixed number of local epochs and sends the updated subnetwork parameters back to the server (algorithm) | line 7). Upon receipt of individual updated subnetwork parameters, the server is then responsible for merging them and assimilating the updates into the shared supernetwork. Critically, it must take into consideration any overlaps or contributions to individual parameters from more than one client (algorithm [ine 1 11-15). Note that FedAvg (McMahan al. 2017) is special case of algorithm [
(App. C where this overlap is constant and equals to the number of clients K. However, the overlap cardinality may vary (E [1, K]) in weight shared FL. This is handled on lines 10-16 16 of algorithm This shared parameeer averaging is one of the key contributions and further investigation opportunity. Now, we describe these two principal framework components in more detail.
2.2 SUBNETWORK DISTRIBUTION
We start by taxonomizing different subnetwork distribution heuristics on the server side. Funda- mentally, the design space for subnetwork distribution heuristics can be divided into two high-level categories: spatial and temporal.
Spatial refers to the distribution of subnetworks among participating clients within a given FL round. Intuitively, this class of heuristics concerns itself with which subnets go where.
Temporal refers the distribution of subnetworks for an individual client across rounds. Intuitively, this class of heuristies concerns itself with when a given subnet forwarded to given client.
Both spatial (which) and temporal (when) aspects should be taken into consideration for subnetwork distribution for best performance. We propose one concrete spatio-temporal heuristic this paper (2.44. Indeed, spatial distribution makes sure that the upper and lower bound of the model family is optimized, while temporal heuristics ensure exposure to all data partitions. The latter can be thought of form of temporal load balancing of individual subnetworks across partitioned datasets. To further validate the importance of spatio-temport heuristics, we perform an ablation study, comparing it with random subnetwork distribution (3.4).
4

---

Under review as a conference paper at ICLR 2023
1: Initialize W
Input: Client j S.) which has been assigned the largest subnetwork rouu (H(()()) archar)
2. for round t 1,2, do
St <- random set of mazeCC K, 1) clients
S. is set of randomly sampled clients 8th round
(St S)
B. decay(Bo,
4: he H(t)) // ttth spatial subnet distr.
a. W zeros(W.))
5:
for client & St do
4: W mam
6
arch <- (k). w < G(W, arch w(+1 ClientUpdaato k, w
Σ
(1-0,)
* n MMWW, archi, wtf1)
1
7:
KES
8: end for
W sum + nn MMW", archs,, w/+1)
W + zeros(W')
largest subnet
I/ shared-param avg by overlap cardinality
W Woum= + replacee_eeeeeeee sraiiee_rrrss(eSen W/) no overlaps
Woum
W t-
ERES, Tey M(W"" archt, wimn) 12. Woum +- replace zeros(W sum. W:)
Σ MMWW arch (1-At) KES
[Sel-1 Tu ones(w)+1))
Wont +
8: Went += 3r * MMW archas n2 * o
Zkes, M(W aachh ny ones(wft1).
9: Went + replace serice__eros(eeeer 1) // no overlaps
14: Went replace_zeros(W6 1)
W
15: W++1 Heam Wont
A Algorithm 2: MaxNet's Shared-Param Averaging For Round f after clients' local updates
end for Algorithm 1: Weight Shared FI. in Server
2.3 SHARED- PAREED-RRRAMTEERR AVERAGING
Shared parameter averaging is the second principal component of the weight shared FL framework. Fundamentally, it provides a scaffolding for implementing heuristics that aggregate parameter updates from participating clients. The server performs distributed subnetworks at the end of each FL round. One naive way of Seared-parameter averaging keep track of the cardinality of the overlap the number of overlaps shared parameters and average the parameters based on this cardinality (Algorithm line 10-16). We call this simplest FedAvg extension to weight shared supernets as averaging by overlap cardinality and analyze it in $3.4 While the framework is designed to accommodate arbitrary spatio-temporally aware aggregation mechanisms, we propose a specific aggregation mechanism in $2.4
paremmter averaging of spatio-temporally
Limitation. In order to fully replace previous round's supernetwork parameters in the current round 6, the largest subnetwork must trained by at least one client. Note that the weights with zero overlap are kept same the previous round's weights (algorithm[]]inn 12 and 14). Since largest subnetwork weights archM are same as supernetwor'' parameters W. the largest subnetwork should be included in each round to ensure the cardinality of overlap 1. Hence, our framework makes simplifying assumption (justified in App. D that spatio-temporal heuristic will sample the largest network each round for least one client.
2.4 SUPERFED'S WEIGHT SHARED FL TRAINING
We propose a novel spatio-temporal subnet distribution heuristic and shared-paramett averaging in this section. We collectively refer to this combination as MaxNet.
MaxNet's subnet distribution. In order to optimize the weight-shared FL objective (App. B equation D we first use the sandwich rule (Yu & Huang 2019b). With the sandwich rule, optimizing the lower and upper bound can implicitly optimize all the architectures in Φ. Hence, if [S,]] is the number of clients participating in the tth FL round, then MaxNet heuristic spatially samples one smallest, one largest and [S]] random subnets. However, not every client participates in even single communication round. The heuristic should train the lower and upper bound on as many clients as possible train muliple model variants. Therefore, temporally, MaxNet keeps track of the number of times the smallest and largest subnetworks have been assigned to each client. Hence, in each communication round, the heuristic assigns the smallest and largest subnetworks to the clients which have them the least. Random subnetworks assigned to the rest of the clients. We call this subnet distribution as Tracking- Bracking-Sandwich. Sandwich.
MaxNet's Mtared-parameed averaging. The shared-paue parameter averaging of MaxNet is derived from averaging by overlap cardinality described in $2.3 We find that naive way of shared- parameeer averaging leads to interference Cai et al. (2020). We report our findings in $3.4 In order reduce
5

---

Under review as a conference paper at ICLR 2023
interference, propose performing weighted average among subnetworks (algorithm) While averaging the parameters of [St] subnetworks, we assign Br (3, E R. 0 < B < 1) weight to the
largest subnetwork's 's parameters and (1-8t
[9,1- weight to the parameters of rest of the subnetworks
(algorithm 2 line 4-5). The training starts with initial value 3. 3. (say 3. 0.9), and 3 value is decayed over some communication rounds to reach BT (say ST = (1 (r))rr)) (algorithm 2 line 2). Our proposed heuristic has three hyperparameters
1) initial weight given to maximum subnetwork in averaging So 2) decay function of 3r 3) decay period of 3r w.r.t percentage of total communication rounds. We perform detailed ablation on these hyperparameters in $3.4 We use So = 0.9 and decay it for 80% of the total communication rounds using cosine decay function as standard hyperparameters for the majority of experiments.
2.4.1 REDUCING INTERFERENCE
Interference is known phenomenon in centralized weight shared training (Cai al., 2020) in which smaller subnetworks interfere with the training of the larger subnetworks. We demonstrate interference in weight shared federated learning ($3.4) and that weight-shared training FL context is non-trivial. We argue that in order to reduce interference, it is paramount to facilitate/priortiz the training of the largest subnetwork first in the global model family. Hence, MaxNet weighs the importance of largest subnetwork's 's parameters and reduces it's influence over time. Specifically, weighted averaging with 3e prioritizes the contributions of the largest subnetwork in the earlier phase of training. We later demonstrate that MaxNet's 's shared-parameter averaging reduces interference among subnetworks (figure 7). In $3.4. we perform a detailed ablation on MaxNet heuristic and its derivations to show attribution of benefit (Tab. 2}.
3 EXPERIMENTS
We show generality of our proposed algorithm on various federated learning scenarios through comprehensive evaluation. We compare SuperFed's MaxNet and independent training (using FedAvg) on the following dimensions (1) real-world datasets (2) non-i..d degree (3) client participation ratio C'. Our goal is to match FedAvg across diverse FL scenarios and achieve an order-of- -fagn-i--a cost reduction w.r.t training global model family. Furthermore, we also provide a cost analysis for global model family training. Later, we perform a detailed ablation study to highlight key factors which contribute our results. We run every experiment for three different seeds and report mean and standard deviation.
3.1 SETUP
Baseline. To our knowledge, there does not exist an FL algorithm which trains a global family of models using weight-sharing. Hence, our baseline is independent SOTA training of DNN subnetworks (without weight sharing) using FedAvg. We call this iFedAvg. While SuperFed trains upto 5 10 subnetworks jointly, it is computationally infeasible to train such a large number of subnetworks with FedAvg independently. We train a family of four DNN models using FedAvg (independently) containing both the smallest (lower bound) and the largest (upper bound) subnetworks of the super- network (App. F Tab. 4). We compare test accuracy/pepplexity of these subnetworks trained using the two approaches and add upto more subnetworks for SuperFed illustrate that SuperFed trains an order of magnitude more subnetworks which is computationally infeasible for iFeeAAg.
Dataset and Models. We compare our algorithm with the baseline on four datasets CIFARIO/10N (Krizhevsky et al., 2009), CINIC-10 (Darlow et al., 2018) and Penn'Tred Brae (PTB) (Marcus et al. 1993). For image datasets, The base supernetwork is ResNet (He al. 2016) based architecture and contains ResNet- 10/26 as the smallest and largest subnetworks respectively. For text dataset, the base supernetwork is TCN (Bai et al., (2088) based architecture (defined in App. H.T). In SuperFed, we train subnetworks varying in the depth and number of filters in convolution layers (App. (F). Heterogeneity in Client Data Similar to (Chen& & Chao, 2020. Lin et al.) 2020a), we use the Dirichlet distribution for disjoint non-iid training data. The parameter controls the degree of heterogeneity a = 100 is a close to uniform distribution of classes. Lower a values increase per-class differences between data partitions (App. KK).
Local Training Hyper-parane We keep client's local training hyperparameters same for SuperFed
and iFedAvg. For image datasets, our training setting is similar to (Lin et al., 2020a) using local

---

Under review as a conference paper at ICLR 2023
0030
0.22
040
143
Fitleg
046
Parte
Moatte (Papuhhl)
040
02
22
922
339
511
021
025
(a) CIFARIO.
(b) CIFARIOO.
(c) CINICIO.
(d) PTB.
Figure 2: Datasets. Test accuracy (higher better) or perplexity (lower is better) comparison between SuperFed and iFedAvg different datasets. training done with 0.4, a 100 and 20/20/110 20 total clients for CIFARIO/ 10/100, CINIC-10 and Penn'reeeBank (PTB) datasets.
SGD (Lin et al. 2020b) with no weight decay and a constant learning rate (no decay). We perform the grid search for learning rate between {0.03, 0.1) on training largest subnetwork using FedAvg (App.K) We find the optimal learning rate to be 0.1 and train the model for 5 local epochs for each communication round". We defer local training hyper-params for text dataset App. H.2
3.2 EVALUATION
We compare our weight shared framework SuperFed with iFedAvg across various federated learning settings in this section.
Accuracy on Real-world Datasets. figure compares MaxNet and iFedAvg on four datasets. For this experiment, we keep C = 0.4 and a 100. CINIC1O/CIFAM 10/100 and PTB experiments are run for 1000 000/11100/2000/ respectively.
communication rounds with total clients as 100/20/20/22
Takeaway. SuperFed's s MaxNet achieves at par (or better) test accuracy/perplexity (for all subuet- works) than iFedAvg. MaxNet gets closer to iFedAvg as the dataset becomes harder (CIFAR 100 is harder dataset than CIFARIO Krizhevsky et al. (2009)).
Effect of Non-ii.d Degree a. figure Bgureeeevllure evaluates MaxNet as the degree of heterogeneity increases. In this experiment. we use CIFARIO dataset with C 0.4 and run different a = {0.1, 1, 100} settings for R 1500 1500 2500 communications rounds respectively.
Takeaway. SuperFed's MaxNet robust to non-i.i.d-ness and experiences no loss of accuracy relative iFedAvg across all the settings. As the degree of non-i.i.d-nes: increases, the difference of the test accuracy between both the approaches decreases.
Effect of Client Participation Ratio C. figure compares SuperFed's 's MaxNet and iFedAvg by varying percentage of clients participating in each communication round. The experiment is run using CIFAR dataset, 100 and 20 clients. Different C 0..2, 0.8) settings are run for R = 1500/ 15000/15///// 11000 communication rounds respectively. Takeaway: MaxNet experiences no loss of accuracy relative to iFedAvg across all the C settings as well. MaxNet's tracking makes sure that every client contributes to the global supernet eventually. Hence, MaxNet is resilient to client participation ratios.
0.90
6.66
Fodlig (Papert)
iFodww
Fidlis
0.75
6.75
Maoher (aog
0.75
Mashel (Ssgerfes)
0.5
1.0
1.5
25
30
35
0.
1.5
1.5
20
30
35
03
10
1.5
22
25
50
35
(a) 100
(b)
(c) 0.1
Figure 3: Non-i.i.d Degree (a). Test accuracy comparison between SuperFed and iFedAvg for different {100, 1.1. values run for 1500/1500/ 1550/25000 comm. rounds respectively. The lower the a value, the heterogeneous the data. Dataset used is CIFARI0 with C 0.4 and 20 clients.
We adopt SOTA approach to batchnorm which is known to be problematic in Fl. non-iid settings (Hsich 2020)
7

---

Under review as a conference paper at ICLR 2023
0.50
0.02
0.80
0.00
6.4
3.00
Tedl
Teat (Spperi)
0.00
0.00
0.00
(SagerTac)
CPLOPs
OFLOPs
GFLOPs
(a) C = 0.2
(b) = 0.4
(c) C = 0.8
Figure 4: Client Participation Ratio (C). Test accuracy comparison between SuperFed and iFedAvg for C {0.2, 0.4, 0.8) values run for 1500/1500/100 1000 FL comm. rounds respectively. Dataset used CIFARIO with a 100 and 20 clients.
0.60
6.82
Too KALRNO
0.90
2.00
0.80
50%
0.68
0.86
0.96
50%
1005
GPLOPs
GTLOPs
(a) 30 value
(b) By-decay function
Decay Period
Figure MaxNet's hyperparameters. a) Initial 8 value used weighted aggregation subnetwork parameters. 8, is assigned to maximum subnetwork and (1- 3. to the rest b) For decaying functions (except constant), 3t is decayed from 0.9 + uniform within the same number of rounds (80% of total rounds) e) Decay period defined % of total rounds until which & decay occurs.
Client's Avg. GFL LOP Round
Avg. GByle Transfer Round
iFedavg
6.32
25
iFedavg
96
MaxNet. (SuperFed)
20
MaxNat (SuperFed)
2.50
10
1:38
5.52
0.67
2.28
Model Family Size
Model FodelFamily Size
(a) Computational Cost
(b) Communication Cost
Figure 6: Casts of training model family in FL. Costs of training model family with varied size is shown. Blue line represents cost of training each subnetwork in the family using FedAvg. The dotted line (orange) line represents costs of training a model family of size Rs 10 using SuperFed. Cost calculation details are described in App.
3.3 COST COMPARISON
now compare SuperFed iFedAvg w.r. computational and communication costs for training a family of models. We defer cost calculation details to App. G
Takeaway. figure shows computational and communication costs of iFedAvg as the model family size increases. The dotted line represents cost for training ~ 5 * 10th subnetworks using SuperFed's MaxNet. Clearly, as the size of model family increases the cost of training model family with iFedAvg approach increases. In fact, training family of 9 models with iFedAvg is 9.43x costlier computationally and 10.94x costlier in communication than training & 5 * 10 subnetworks using SuperFed's MaxNet. Hence, iFedAvg is infeasible if used for training subnetworks of the same order of magnitude as SuperFed. Furthermore, we emphasize that SuperFed incurs smaller cost than independently training the largest network with iFedAvg (model family size as App. EE). making the preferred method of federated training for any family of models, however small.
3.4 ABLATION STUDY
We now study attribution of benefits by performing an ablation on 1) Spatio Temporal Heuristics 2) Hyperparameters of MaxNet heuristic.
8

---

Under review as a conference paper at ICLR 2023
0.92
Apcaneyy
0.90
Heuristie
Subnetwokk
Averaging
0.60
overlap
overlap
ity
overlap 8
Sundwich
everlap cardinal-
ity
0.64
(watip 4
morii TB
VRl Frdicay
TS (aaesil
overlap + IS
Tracking- Sandwich
ity
overlap cardinal-
(-docny
Tracking-
Wi (odeeay
0.5
1.0
2.0
2.5
(MaNNet)
Sandwich
overlap cardinal-
ity
Figure 7: SuperFed's Spatio- Temporal Heuris- tics. Test accuracy comparison for various spatio- temporal heuristics. The dataset used is CIFAR1O with C = 0.4, a = 100 and 20 clients.
Table 2: Description of subnetwork distribution and shared-parameter averaging for different spatio-temporal subnetwork distributions SuperFed
Spatio Temporal Heuristics. Tab, lists the spatio-temporal heuristics tried as a part of this ablation study. For fair comparison, all the training hyperparameters are kept constant CIFARIO dataset, === 0.4 and a = 100. figure 7 compares the test accuracy of all the spatio-temporal heuristics. Takeaway. First, MaxNet has the best accuracy for all the subnetworks among all other heuristics. Second, notice that the simple overlap cardinality averaging (overlap + R) with random subnetwork distribution underperforms for all subnetworks compared to independently training them. While, both sandwich (overlap + S) and tracking-sandwich sandwich (overlap TS) subnetwork distribution improve smallest subnetwork's 's accuracy as compared to random subnetwork distribution (overlap + R) and match the accuracy of independently training it. Tracking-sandwich sandwich (overlap TS) performs slightly better than sandwich (overlap + S). However, the largest subnetwork's accuracy is still sub-par compared to iFedAvg. We attribute the reason for sub-par accuracy of the largest subnetwork to interference (for overlap + (R,S,TS) SS) in figure 77. i.e. smaller subnetworks interfere with the training of the larger subnetworks Cai et al. (2020). MaxNet prioritizes contributions of the largest subnetwork in the earlier phase of training and decays its importance over time. This reduces interference and achieves no loss of accuracy compared independently trained subnetworks.
Hy plpeepaarmeters of MaxNet. MaxNet's hyperparameters are described in $2.4 We perform grid search on the following hyperparameters: Bo {0.9, 0.5) decay function [linear, cosine, constant), decay period (50%, 80%). S test accuracy of the subnetworks for different hyperparameters.
Takeaway. Hyperparameter 3o has a major effect on test accuracy of subnetworks (figure 5a). Starting the training with larger 8g value helps reduce interference. Hence, training of the largest subnetwork is preferred more in the earlier phase of training, which also benefits the smaller networks. Also, keeping 3, = 0.9 const only improves largest subnetwork's accuracy. Decaying 3, over communication rounds increases accuracy for smaller subnetworks without reducing the largest subnetwork accuracy(figur) (5).. Similar empirical evidence of the benefit of gradually reducing larger subnetwork's 's importance in training also seen in OFA (Cai et al. (0200). Moreover, cosine decay performs better than linear decay. Longer decay period (80% of the total comm. rounds) further helps in achieving better test accuracy (figure 5c
CONCLUSION
SuperFed the new framework for weight shared FL. Its pluggable architecture is used to evaluate newly proposed set of distributed training algorithms for spatio-temporal subnetwork distribution and model parameter aggregation, collectively called MaxNet. MaxNet is able to train multiple DNNs jointly with weight sharing in a federated fashion, matching ofute-of-the-ar FedAvg w.r.t accuracy. SuperFed amortizes the cost of training over an arbitrary selection of subnetworks, while conventional FL. techniques (e.g., FedAvg) incur (kk) training cost for any selection of & models from the "family of models". SuperFed opens new directions for future work: adjusting MaxNet for client resource constraints (e.g., compute, bandwidth, memory capacity) and developing theoretical insights on training enabled by MaxNet. We believe that SuperFed takes an important first step towards co-training model families in federated fashion cost-efficiently, achieving an order of magnitude communication and computation cost reduction (even with as few as k = 9 models) while maintaining accuracy through interference mitigation.

---

Under review as a conference paper at ICLR 2023
5 REPRODUCIBILITY STATEMENT
We provide anonymized source code https: hteps://anonymous h 4open.sciences scw.ece/rr./eer..r SuperFed/ for the experimental results included in the main text. The code includes full instructions to perform weight-shared federated learning on the datasets and settings considered in the paper.
REFERENCES
Apple. designing for privacy (video and slide deck). apple wwde, 2019. URL https://
deve Developer.oo appla com/videos http///ww221//////000000
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. International Conference Learning Representations, 2021. http://pps//oeeeeeeeeeeeeeeeeeee
id=B7v4QMR629w
Alexa. Amazon alexa. 09-18.
htt.:///....aaaa..ooooooooooo//a Accessed: 2022-
Dan Alistarh, Demjan Grubic, Jerry Z. Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-ffiiiien sgd via gradient quantization and encoding. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 1707-1718, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN
Muhammad Ammad-ud-din. Elena Ivannikova, Suleiman A. Khan, Were Oyomno, Qiang Fu, Kuan Eeik Tan, and Adrian Flanagan. Federated collaborative filtering for privacy-preserving personalized recommendation system. CoRR, abs/1901,094 00888, 2019. URL http://arxiv erq/abs/ 1901. 09888
apple neural engine.
Deploying transformers on the apple neural engine. mmachineeearning http://gw..pppppeeeee/eeeeeee/aa cessed: 2022-09-18.
https:/ //
toa.-ne-ttraaffooeeee... Ac-
azure vision. Azure computer vision. wtt.in//ttsss/aaeuee.lcorooooff
Accessed: 2022-09-18.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional
and recurrent networks for sequence modeling. CoRR, abs//803.01271, 2018. URL http:
/ //arxiv... wrg. abs/ 1803. 10.00.0122 01271
Neelkamal Bhuyan and Sharayu Moharir. multi-model federated learning. In 2022 14th International Conference on COMmunication Systems & NETworkS (COMSNETS), pp. 779-783. IEEE, 2022.
Chenzhong Bin, Tianlong Gu, Yanpeng Sun, Liang Chang, and Lei Sun. A travel route recommenda- tion system based on smart phones and iot environment. Wireless Communications and Mobile Computing, 2019, 2019.
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Konečný. Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: System design. A. Talwalkar, V. Smith, and M. Zaharia (eds.), Proceedings of Machine Learning and Systems, volume pp. 374-388, 2019. URL https:ttp...mmmm //tp:c/ee///ceeeppeeoooooo ngs. mm..mmm..... .myysss...................mm Papeerapppppp..
org/ arco//aperr/211//11e//aaaae
4 38 http////0......a..........0.a.a0
Ermao Cai, Da- Pa-Cheng Juan, Dimitrios Stamoulis, and Diana Marculescu. Neuralpower: Predict and
deploy energy-efficiem convolutional neural networks. In Asian Conference on Machine Learning, pp. 622-637. PMLR, 2017.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize for efficient deployment. In International Conference on Learning Representations, 2020. / wwt.:eww...e.....eeeeeeeeeee wtt.:eewew...ooommmmmnm.y...
10

---

Under review as a conference paper at ICLR 2023
Hong- You Chen and Wei-Lun Chao. Feddistill: Making bayesian model ensemble applicable to federated learning. CoRR, abs/2009.01974, 2020. URL [ttlp::////a/ii.... http//twwii...r//b...//a 2009
01974
Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy, Tym: An automated end-to-end optimizing compiler for deep learning. In Proceedings of the 13th USENIX Conference Operating Systems Design and Implementation, OSDI 18, pp. 579-594, USA, 2018.
USENIX Association. ISBN 0781931971476
EU CORDIS.
Machine learning ledger orchestration for drug discovery, 2019. mc RSS Feed&
URL
f=pro rss -le_a_a222334w rss_ rss_ev=a ev=a
Rafael Valente da Silva, Jinho Choi, Jihong Park, Glauber Brante, and Richard Demo Souza. Multi- channel aloha optimization for federated learning with multiple models. IEEE Wireless Communi- cations Letters, 2022.
Luke Nicholas Darlow, Elliot J. Crowley, Antreas Antoniou, and Amos J. Storkey. CINIC-10 is not imagenet or CIFAR-10. CoRR, abs/1810.03505, 2018. /tLLitttpp//axxxxooo///////
1810. 03505
Sauptik Dhar, Junyao Guo, Jiayi (Jason) Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah. A survey of on-device machine learning: An algorithms and learning theory perspective. ACM Trans. Internet Things, 2(3), jul 2021. ISSN 2691-1914. doi: 10.1145/3450494, URL https
//doi. 1441344444445444
Enmao Diao, Jie Ding. and Vahid Tarokh. Heterofl: Computation and communication efficient
federated learning for heterogeneous clients. CoRR, abs/2010.01264, 2020. URL https://
arxiv. Ltxiv.org/abs//0111 01264
Biyi Fang, Xiao Zeng, and Mi Zhang. Nestdnn: Resource- multi-tenant on-device deep learning for continuous mobile vision. CoRR, abs/1810. 10090, 2018. UK./Ectp://axx/w.o////bbb
1810 10090
Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Doermann, and Arun Innanje. Ensemble attention distillation for Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15076-15086, 2021.
federated learning. In
google ASR. Google automatic speech recognition. Speech-to-ten tenech-to-ee Accessed: 2022-09-18.
https://elou..gooiies com/
google vision. Google cloud vision ai. 2022-09-18.
Accessed:
Myeonggyun Han, Jihoon Hyun, Seongbeom Park, Jinsu Park, and Woongki Baek. Mosaic: Heterogeneity- communication- and constraint-aware model slicing and execution for accu- rate and efficient inference. In 2019 28th International Conference on Parallel Architectures and Compilation Techniques (PACT), pp. 177- IEEE, 2019.
Andrew Hard, Chloé M Kiddon, Daniel Ramage, Francoise Beaufays, Hubert Eichner, Kanishka Rao, Rajiv Mathews, and Sean Augenstein. Federated learning for mobile keyboard prediction, 2018. URL httthhtpt///aait
org/ abs/1817 03604
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Fednas: Federated deep learning via neural architecture search. CoRR, abs/2004.08544 2020. URL httptpts///aiiiiorgg/////// 2004. 08546
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
11

---

Under review as a conference paper at ICLR 2023
A. Howard, M. Sandler, B. Chen, W. Wang, L. Chen, M. Tan, G. Chu, V. Vasudevan, Y. Zhu, R. Pang, H. Adam, and Q. Le. Searching for mobilenetv3. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1314-1324, Los Alamitos, CA, USA, nov 2019. IEEE Computer Society. doi: 10. 20.100//CCCV22219.00000 000400 URL httpts:/d// i Teeecomuteersoc iety 110.110.1119100 100..1119.CC...
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-llID data quagmire of decentralized machine learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4387 PMLR, 13-18 Jul 2020. URL httpp::/pprcceeeeheee
mlr
Li Huang and Dianbo Liu. Patient clustering improves efficiency of federated machine learning to predict mortality and hospital stay time using distributed electronic medical records. CoRR, abs/1903.09296 2019.
09296
Nikita Ivkin, Daniel Rothchild, Enayat Ullah, Vladimir Braverman, Ion Stoica, and Raman Arora. Communication Effiicient Distributed SGD with Sketching. Curran Associates Inc., Red Hook, NY, USA, 2019.
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi. Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage. Ramesh Raskar, Dawn Song. Weikang Song, Sehastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang. Li Xiong, Zheng Xu, Qiang Yang, Felix Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning, 2021.
Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. ACM SIGARCH Computer Architecture News, 45(1):615-629, 2017.
Sai Pranceth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theerthh Suresh. Scaffold: Stochastic controlled averaging for federated learning, 2021.
Mikhail Khodak, Renbo Tu, Tian Li, Liam Li, Maria-Florina Balcan, Virginia Smith, and Ameet Talwalkar. Federated hyperparameter tuning: Challenges, baselines, and connections to weight- sharing. CORR, abs/2106.0ee 0440050 2021.
2tt.://////21........ 04502
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárík, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 2016.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Nicholas D Lane, Sourav Bhattacharya, Petko Georgiev, Claudio Forlivesi, Lei Jiao, Lorena Qendro, and Fahim Kawsar. Deepx: A software accelerator for low-power deep learning inference on mobile devices. In 2016 15th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), pp. 1-12. IEEF, 2016
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks, 2020.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model
fusion in federated learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin
(eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2351 2351-2363. Curran As- sociates, Inc., 2020a. URL /ttpo///pepppp////o//occooooooo http://cweeiiine..ie......o..see www.wnnuuuicccmm http:///ww.e.aeiieeeeecc///////a /220//i11////////////// file/ T
12

---

Under review as a conference paper at ICLR 2023
Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batches, use local sgd. In International Conference on Learning Representations, 2020b. URL https
net/ //ppee/mssie....e///ooooo.taaaaa Btt.:/rrmmmmmm11m111111
Mitchell P. Marcus, Mary Ann Mareinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. Comput. Linguist., 19():313-330, jun 1993. ISSN 0891-2017.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-cficce learning deep networks from decentralized data, 2017.
Fan Mo. Hamed Haddadi, Kleomenis Katevas, Eduard Marin, Diego Perino, and Nicolas Kourtellis. Ppfl: privacy-preserving federated learning with trusted execution environments. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services, pp- 94-108, 2021.
Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, and Carole-Jean Wu. The vision behind miperf: Understanding inference performance. IEEE Micro, 41(3):10-18, 2021.
Manas Sahni, Shreya Varshini, Alind Khare, and Alexey Tumanov. Comp|ofa| compound once- for-all networks for faster multi pultti-platfomm deployment. International Conference on Learning Representations, 2021. URL
samsung exynos. Samsung exynos 9825. htt:s:://smmcooddccttt samsung. com/
processor wwteesssrmmoobieee. prwcessor exynos 9825/ Accessed: 2022-09-18.
Santiago Silva, Boris A. Gutman, Eduardo Romero, Paul M. Thompson, Andre Altmann, and Marco Lorenzi. Federated learning in distributed medical databases: Meta-analysis of large-scale subcortical brain data. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI doi: 10.1 10111091..010..8755555
2019), pp. 270-274, 2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task-agnostic BERT for resource-limited devices. CoRR, abs/2004.02984, 2020. URL /tt::///axxivi http://rxxxv..rr///////0000..0.
torch mobile. Pytorch mobile. https: 18.
Accessed: 2022-09-
Stacey Truex, Nathalic Baracaldo, Ali Anwar, Thomas Steinke, Heiko Ludwig, Rui Zhang, and Yi Zhou. A hybrid approach to privacy-preee prrvacy-rrssering federated learning. In Proceedings of the 12th ACM workshop on artificial intelligence and security, pp. 1-11, 2019.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Fed- crated learning with matched averaging. In International Conference on Learning Representations, 2020. URLhttps: http://::////eeeeeeeeeeee net wwt.:/eetetorrrrm.m/mm
forum? Ltttt//oorummaaBBLuulll
Bichen WU, Xiaoliang Dai, Peizhao Zhang. Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In IEEE Conference Computer Vision
and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 10734-
10742. Computer Vision Foundation / IEEE, 2019a. doi: 10.1109/CVPR.200 201.PP...01900999990 URL
http://w...ooooooooooet/ee CVPR CECR_CCPP111000C
Hardware-Aware wtwaaarre__ffiiieeee wwwp:eetttttoooeeeeeeeeee SerrrWeee_Deeiig_Weee ot www.eeet.eeeeeiaallleeee
Neural hitaaaaacccaeecturreeeee.......n
Carole-Jean Wu, David Brooks, Kevin Chen, Douglas Chen, Sy Choudhury, Marat Dukhan, Kim Hazelwood, Eldad Isaac, Yangqing Jia, Bill Jia, et al. Machine learning facebook: Understanding inference at the edge. In 20/9 IEEE international symposium on high performance computer architecture (HPCA), pp. 331-344. IEEE, 2019b.
Jin Xu, Xu Tan, Kaitao Song, Renqian Luo, Yichong Leng, Tao Qin, Tie- Yan Liu, and Jian Li. Analyzing and mitigating interference in neural architecture search. In International Conference Machine Learning, pp. 24646-24662. PMLR, 2022.
13

---

Under review as a conference paper at ICLR 2023
Zirui Xu, Fuxun Yu, Chenchen Liu, and Xiang Chen. Reform: Static and dynamic resource-
aware dnn reconfiguration framework for mobile device. In Proceedings of the 56th Annual
Design Automation Conference 2019, DAC '19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367257. doi: 10.1 10.1145/331678 1114//3316788.33344666 URL https; 3324669
7/doi. .rg. 10.
Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1803-1811, 2019a.
Jiahui Yu and Thomas S. Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019b.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference on Learning Representations, 2019. URL htttp:://openrevieew
net / wow.:miinddll.c...a...
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architec- ture search with big single-stage models. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 702-717, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58571-6
14

---

Under review as a conference paper at ICLR 2023
A BACKGROUND ON WEIGHT-SHARED TRAINING
OFA (Cai et al. 2020) proposes progressive shrinking method for weight-shared training of 1019 subnetworks jointly by using a neural supernetwork. The training is facilitated by pre-trained teacher network which is used for distillation throughout the training. Training a supernetwork naively causes "interference" (Cai et al. 2220)). Hence, OFA takes a phased approach to training, by training larger subnetworks first and progressively shrinking the dimensions of a subnetwork being trained. BigNAS (Yu et al. 2020) proposes one-shot approach (doesn't 't any pretrained network) for training a supernetwork using inplace distillation (Yu & Huang, 2019a). also proposes several training hyperparameters which optimize the supemetwork training. However, these weight-shared training techniques are proposed for centralised setting with i.i data. The proposed SuperFed is a one-shot weight shared training framework for non-i.i.d decentralised data. We empirically show that naive application of FL techniques for globally training weight shared supernetwork structures causes interference among subnetworks in non-i.i.d setting ($3.4).
B PROBLEM FORMULATION
Now, we formulate the objective for our weight shared federated learning (notations listed in App. D Tab. (3. Later, we discuss our proposed heuristic to minimize the objective. FedAvg (McMahan al. 2017), single global model with parameters is trained by minimizing the following objective function min,, v: Σk
Ex-i x Fx (w). where there are K clients with their own data partition Ps. such
that the size of the partition is denoted by n& (where n Σx= n& is the total number of data points). (W)) = Eiep, fi(w) denotes the loss for data points of partition Ps
Meanwhile, in weight shared training, OFA's goal is to train every possible architecture in the supernetwork. Assume o {archı ,arch2 archas } as a DNN model family with archas being the largest subnetwork. The shared weights of these architectures are denoted by W (su- pernet's 's parameters). Hence the objective OFA (Cai et al., 2020) can be formalized as follows: minw Larch, (G,, arch,))). where G(W, arch,) denotes a selection (subset) of arch; parameters from shared weights W. L(w) denotes the loss of subnetwork's s parameters (w E W) on a single central dataset.
As stated in $1 the goal of this work is to train global model family of size (Ø). Thus, the objective function of such training takes the following form:
min Σ
The n
* Fe(G arch)))
Indeed, the goal of weight shared FI. training is to train all the architectures in the family & on all the data partitions (P1, P2..... Px); Weight shared FL aims training a family of models & at order of magnitude lesser training cost (both communication and computational) relative to independently training these networks in FL while matching or even surpassing accuracy. We compare weight
shared FL trained subnetworks with independently FL trained subnetworks in 3.2
C FEDAVG: A SPECIAL CASE OF SUPERFED
FedAvg (McMahan et al., 2017) is a special case of SuperFed's algorithm (algorithm I. When the family of models contains only a single subnetwork i.e. ő {arch). then ] only trains the weights of that subnetwork. The spatio-temporal heuristic 7. becomes (k) arch Vt e (1,2,. J. & E S. Since the same subnetwork is sent to all the clients participating in round, the number of overlaps for weights of subnetwork arch equals the number of participating clients |S,]. Hence the averaging method reduces to FedAvg's averaging mechanism for subnetwork arch.
15

---

Under review as a conference paper at ICLR 2023
D DESCRIPTION OF NOTATIONS
Table 3: Notations used in the paper
Notation
Explanation
K
The number of clients
The set of clients involved in FL training, S = {1, 21, K
T
The total number of communication rounds in
Pr
Dataset partition for client &
Sum of loss of data points in Ps for DNN weight Size of dataset partition Ps: ([P&)
ne
C
Client Participation ratio in federated setting
Family of DNN models or set of DNN architectures Weight of supernetwork
w
Weight of subnetwork contained inside supernetwork DNN architecture (subnetwork) such that arch E Φ Selection of weights for DNN arch from supernet weights W
arch
arch
largest subnetwork in e such that G(W, (Wrchar) = WM =
h
Spatial heuristic for subnetwork distribution such that h S ->
H(t))
Spatio-temporal heuristic for subnetwork distribution such that H: Weight of subnetwork arch partially replacing weight W of supernetwork (super-imposition)
-
M(W, arch, w)
E COST COMPARISON FOR TRAINING THE LARGEST SUBNETWORK
Since supereewoor's 's weights are same the largest subnetwork's 's weights, compare SuperFed and FedAvg w.r.t communication and computational costs of training the same weights for total communication rounds T. In SuperFed, other subnetworks (i.e., subsets of the largest subnetwork parameters) are also trained along with the largest subnetwork while FedAvg only trains the largest subnetwork. We assume that the largest subnetwork is assigned to atleast in SuperFed (assumption in 2.3).. Note that the overall time spent by a client in training the neural network(s) send by the server over multiple rounds would be less in SuperFed than FedAvg. Moreover, the amount of bytes exchanged between the server and clients would be less in SuperFed than FedAvg in every round. This is because in SuperFed, smaller subnetworks are also sent to some clients other than just sending the largest subnetwork to every client (FedAvg). These smaller subnetworks not only take less training time (computational) but also have less parameters (communication). Hence, proposed weight shared federated learning framework trains family of models with lesser computational and communication costs than training the largest subnetwork using FedAvg.
client for every round
F DNN ARCHITECTURE SPACE AND SELECTION FOR IMAGE DATASETS
We modified OFAResNets - a supernetwork proposed in OFA (Cai et al. 2020) to have smaller size (10-26 layers) models and make it amenable 32.32 image size. A typical ResNet architecture He et al. (2016) has multiple stages and each stage contains list of residual blocks. We implement the supernetwork with 4 such stages. Next, we describe the two elastic dimensions use vary subnetworks in size.
Depth. It decides the no. of residual blocks per stage. Hence, this dimension is usually specified using list of size 4 with each value denoting the depth of that stage. Our smallest subnetwork keeps depth of two per stage. Hence these two per-stage residual blocks are shared among every subnetwork. Depth elastic dimension can also be specified using single integer say d which denotes depth for every stage in the supernetwork.
Elastic ratio. This dimension changes number of filters and channels of convolution layers in residual block. The block has two convolutional layers with input, middle and output channels. Elastic ratio is ratio of output middle channel only affecting the number middle channels in the block. The elastic ratio is specified via list of size 12 expressing number of middle channels for each residual block. It can also be specified using single integer say which denotes elastic ratio for
16

---

Under review as a conference paper at ICLR 2023
every residual block in the supemeework.
Tab. lists the subnetworks selected for comparing SuperFed with iFedAvg. Out of the subnetworks, 4 of them are also trained using FedAvg independently, SuperFed s 5 10 subnetworks including the nine subnetworks listed Tab.
Table 4: Dimensions of subnetwork architectures shown in all the figures. Out of nine architectures, four of them are also trained using FedAvg independently for comparison.
Subnetwork Depth Subnetwork Expand Ratio
GB
FedAvg Model Family
[0,0,0,0]
0,1
0.40
[0,0,0,1]
0.14
0.83
[0,1,0,1]
0.14
0.85
[0,1.1.1]
0.14
0.94
[1,1.1.1]
0.18
1.17
[1,1,1.2]
0.22
1.91
[1,2,1,2]
0.22
1.94
[1,2,2,2]
0.22
2.08
[2,2,,22]
0.25
2.36
G COST CALCULATION
G.1 COMPUTATIONAL COST
We define computational cost the average time spent/computations done by the client in FL training. Hence, we estimate average computational cost in single FL communication round. Note that the cost proportional the GFLOPs of the model (subnetwork) that training. Now, we describe cost calculation for the two approaches compared in [
SuperFed. Sum GFLOPs of all the subnetworks that each clients sees over the course of training
procedure is done. Then, an average is calculated over the total number of clients and total number of communication rounds.
ivedAvg. Here all the subnetworks of a model family trained independently. The GFLOPs of subnetwork is proportional to the time client spends per round in FedAvg training Hence, the sum of GFLOPs of all subnetworks in the family is the average computational cost per round in iFedAvg.
G.2 COMMUNICATION COST
Communication cost is defined by the average bytes transferred between the clients and the server in FL training. The total communication cost is the sum communication cost per round. Hence, we first calculate average communication cost per FL round. The average includes both the download and upload of the model. The bytes transferred to the clients and vice-versa depend on the size of the subnetwork. Now we describe communication cost calculation
SuperFed. The sum size (in GB) of subnetworks distributed to each client at each communication round is calculated. Then average is taken over total communication round.
AvvdAAgg We first calculate average communicaton cost per round for training a single subnetwork (say arch:). This is typically given by 2 [S, sizeOf(arch,) ([S& denotes number of participating clients). Since every subnetwork is trained independently the total communication cost per round is given by- 2 |S&| Earch.eo sizeOf(arch.). In figure 6 we show costs for training model family of size upto nine subnetworks with iFedAvg.
H DNN ARCHITECTURE SPACE AND CLIENT LOCAL TRAINING FOR TEXT DATASET
In this section, we describe our TCN Bai et al. (2018) supernetwork and local training hyper- parameters associated with the text dataset experiment.
17

---

Under review as a conference paper at ICLR 2023
H.1 SUPERNET BASED ON TCNS
A standard TCN architecture (Bai et al. 2018) for PTB dataset Marcus al. (1993) consists of eight layer (or four temporal blocks) with 600 input and output channels. We enable two clastic dimensions in the standard TCN architecture
depth and expand ratio. Depth defines the the number of blocks to be activated whereas the expand ratio defines the number output channels the first layer of each block. Overall, the depth choices considered are (0,1,2) with denoting activating top two blocks only and expand ratio choices are {0.1, 0.2, 0.5, 1.0).
H.2 LOCAL TRAINING HYPER-PARAMS PARAMS
For client local training in text dataset, the initial learning rate is kept as 4 which is a standard learning rate train TCN on PTB dataset. We decay the learning rate every 50 local epochs by 0.1. We use gradient norm clipping as 0.35 and batch size 16. We run five local epochs per client for every communication round
I NEURAL ARCHITECTURE SEARCH
Table NAS performed for 3 GFLOPs constraints using seed 0 best model checkpoint for cifar10
alpha 100.
NAS Constraint (GFLOPS)
Test Accuracy
GFLOPs
Subnelwork Architecture
89.98
0.9
0.1.1 18, 0.22 O.T. O.TK 0.14, 0.22 0.T8 O.T. 0.22 0,22, 0.14 411,220
d:1.00000))
91.25
139
0.2 0.2. 22 O. L.T. O.E. U.22 O. I.I 0.18, 0.25, 0.18
91.33
272
2122222
0.25, D22 0.14 D. 0.25, 0.25, 0.25, D.T. 6.1]
The goal of weight shared FL is offer rich latency accuracy trade-off space so that clients can choose one more models for her deployment needs. In this section, we demonstrate that once the supernetwork is trained, a neural architectural search (NAS) can be performed by the clients to extract Pareto-optimal subnetworks. Similar to Sahni et al. (2021), we use evolutionary algorithm to search for pareto-optimal subnetworks for a given FLOP constraint. Tab, Tab,55lists different subnetworks found by the search algorithm for different GFLOPs constraints. The subnetworks extracted from supernetwork trained with the following setting C 0.4, a = 100, 20 clients and CIFAR-10 dataset. The subnetworks are extracted without retraining.
J JUSTIFICATION OF SUPERFED'S LIMITATION
We first highlight that goal SuperFed is to target multiple deployment scenarios of clients rather than assume system-heterogeneiiyy during training. Furthermore, we argue that resources are extremely dynamic and scarce in deployment (discussed in { than during Fl.-training. One of the foremost requirement of FL is to make the model training invisible the user i.c. it should not slow down the smartphone or drain its battery (Bonawitz et al., 2019). Hence, smartphones usually participate in FL-training when they are idle, above certain battery level and connected to unmetered network (Kairouz et al.) 2021). Since resources are fairly available at FL-training time, it is reasonable to assume that atleast one client takes the largest subnetwork during FL-training given comm. round (a limitation discussed in 2.3).
K LEARNING RATE GRID SEARCH
We perform a grid search on learning rate for the largest subnetwork in the model family using FedAvg. figure g compares learning rates 0.03 and 0.1. This shows that learning rate 0.1 achieves better test accuracy for 1500 communication round. To have less complexity, we keep the same learning rate for training each subnetwork using FedAvg. SuperFed also uses learning rate as 0.1 for fair comparison.
18

---

Under review as a conference paper at ICLR 2023
Accuracy
0.4
R=0.1
LR-0.03
250
500
750
1000
1250
1500
Communication Round
Figure 8: LR Grid Search. Comparison of test accuracy for two different learning rate values {0.03, 0.1 on CIFAR-10 dataset using FedAvg. The total number of clients are 20 with C' 0.4 and 100.
CLIENT CLASS DISTRIBUTION
6
Class 6
Class
2
12 14 16 18
6
18 18
6
Client ID
Client
ID
Client ID
(a) 100.
(b)
(c) a
Figure 9: Non-i..d Degree (o)). Client class distribution visualization across different {100. 0.1}
values. Showing visualizations for CIFARIO dataset partitioned using Drichlet distribution among 20 clients.
figure visualizes distribution of classes among 20 clients. The size dots symbolizes the amount of data points for that particular class in client's dataset partition. Partition is done using dirichlet distribution which is parameterized by a. As seen in the figure, the lower the a value the higher the degree of non-i.i.-nness. Specifically for 0.1, many clients don't have even single data point for some classes.
19


Reviews:
[{"review_id": "SyCuJDMZMZ", "reviewer": "Reviewer_nfyJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a weight shared training framework for FL that trains a family of model variants (DNN models), cost-efficiently in a federated fashion. The paper also proposes a heuristic that trains the upper and lower bounds in the model family by optimizing both bounds and load balancing their distribution over time. The paper further proposes a mechanism for subnetwork weight aggregation with variable overlaps mitigating intra-network interference. The proposed approach can take an important first step towards co-training model families in a federated fashion cost-efficiently, achieving an order of magnitude communication and computation cost reduction.", "summary_of_the_review": "The paper studies a very novel problem setting 'training a global family of models'. However, maintaining many models in a real-world applications can introduce a lot of complexities. Not sure whether it is practical in the real world. Currently there is only one baseline evaluated in the paper. It is understandable that the paper proposes a very novel setting ('training a global family of models'), so the comparable baselines are limited. However, we may still align the setting and compare the proposed approach to existing FL approaches.", "strength_and_weaknesses": "Strength:\n1) The proposed concept of 'training a global family of models' seems to be very well-motivated, novel and appealing.\n2) The proposed method clearly outperforms the baseline, independent SOTA training of DNN subnetworks (without weight sharing) using FedAvg.\n\nWeakness:\n1) The concept of 'training a global family of models' is well-motivated by the need in on-device inference in real-world applications. However, by multiple model variants, I assume the variances of the performances in each devices can be much more significant, compared to the single model in previous FL approaches. However, only achieving some level of accuracy is not enough to release a real product (e.g, many human evaluations of the user experience are needed). With many different models in many devices, it will incur a lot of efforts to track and maintain these models. Thus, I am not sure whether the concept of 'training a global family of models' is practical in real-world applications.\n2) Currently there is only one baseline evaluated in the paper. It is understandable that the paper proposes a very novel setting ('training a global family of models'), so the comparable baselines are limited. However, we may still align the setting and compare the proposed approach to existing FL approaches? For example, assume there are 10 models. For each client, we can train a suitable model by existing FL approaches on all clients with similar (or less) devices constraints. Although there are some benefits of 'training a global family of models', these experiments can still provide some insights about whether the proposed solution outperforms SOTA FL approaches (or still has a gap) in FL problems (e.g., non-iid).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "H_mCx8ymvu", "reviewer": "Reviewer_uEYf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a federated learning method that supports training multiple models in various sizes simultaneously without heavy costs and performance loss. The core idea is weight sharing, where different models can share common knowledge. Experimental results show the effectiveness of the proposed method.", "summary_of_the_review": "This work is interesting but its insufficient clarity is a major concern. The technical contribution seems to be not very big. Thus, my recommendation is \"marginally below the acceptance threshold\".", "strength_and_weaknesses": "Strengths:\n1. The problem studied in this paper is interesting.\n2. The proposed method seems to be effective.\n3. The concept of temporal-spatial subnetwork distribution is interesting.\n\nWeaknesses:\n1. The relations between this work and existing client-heterogeneous FL works [1-3] are not clarified ([2] and [3] are also not included in this paper). The authors mention that \"Diao et al. (2020); Lin et al. (2020a) provide a single global model to every client\", which is quite confusing because Diao et al. (2020) distribute different models to different clients. In fact, these methods can choose to distribute models in different sizes to different clients in different rounds. Moreover, [1] and [3] also consider weight sharing mechanisms. Thus, it is not clear what is the difference between the scenario in this paper and existing works, as well as the unique advantage of the proposed method.\n2. The authors state that \"To our knowledge, there does not exist an FL algorithm which trains a global family of\nmodels using weight-sharing\". This may not be the case. Thus, more comparisons with existing baselines are needed.\n3. The methodology section is somewhat too brief, and many details are not well clarified. \n\n\n[1] HeteroFL: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264.\n[2] FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization. arXiv preprint arXiv:2111.14655.\n[3] No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices. KDD 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9gtecbHEk2", "reviewer": "Reviewer_EQW3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a weight shared training framework for FL that maintains a primary network at the server and distributes subnetworks to clients. The authors propose algorithms for efficient selection of the subnetworks and aggregation. This allows efficient federated learning and deployment of the models. \n", "summary_of_the_review": "The paper provides an interesting and relevant problem setting and an effective algorithm for this setting. Validation on additional datasets and broad model classes would  increase the strength of the work.", "strength_and_weaknesses": "Strengths\n\n- The proposed problem (not directly studied in other works) is relevant\n- the approach is relatively straightforward and elegant\n- The results are convincing and well analyzed on the presented datasets\n\nWeakness\n- Although the datasets are standard for FL the authors primarily focus on variants of CIFAR-10 and convnets. It would be beneficial  to see the generality of the approach to other architectures (e.g. MLP's and ViT's) and more distinct datasets (e.g. text data). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4517H5zAbKh", "reviewer": "Reviewer_8Rfx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a weight-sharing approach for simultaneously training multiple models with different architectures to satisfy different deployment needs in a federated setting. Their method combines existing techniques for weight-sharing with new approaches for distributing architectures and averaging parameters. The resulting algorithm is evaluated on three vision datasets. Note that I have reviewed this paper before and parts of my review are taken from my previous evaluation.", "summary_of_the_review": "The problem addressed is well-motivated and the authors obtain reasonable empirical results on the datasets considered, but the evaluation remains limited to CIFAR-like datasets, which makes the applicability to practical FL unclear.", "strength_and_weaknesses": "### Strengths:\n1. The problem setting is well-motivated and a practical algorithm here would be valuable for applications of federated learning.\n2. The authors conduct ablation studies of their solutions for handling interference due to weight-sharing.\n3. Experimentally there is a comparison to reasonable baselines (iFedAvg) and the authors demonstrate improvement by the method in obtaining a superior accuracy-complexity tradeoff. There is also evaluation of the method at different heterogeneity levels, albeit synthetically generated via classes.\n\n### Weaknesses:\n1. The main weakness of the paper remains the limited evaluation. The proposed algorithm is evaluated on three closely related datasets, all containing natural images with the same size. Most FL papers will at least include a text application alongside evaluation on image data, given that text is one of the dominant applications of FL. There is not a strong reason to know that the proposed method will directly extend beyond image data, especially since the non-FL work this paper is based on (OFA) focuses on vision. I believe to be impactful the paper would at least have to evaluate on one of the text datasets in LEAF (Caldas et al., 2019), or on more recent datasets based on Reddit or StackOverflow.\n2. It is unclear if code will be provided for reproducibility.\n\n### References:\n1. Caldas, Duddu, Wu, Li, Konecny, McMahan, Smith, Talwalkar. LEAF: A benchmark for federated settings. arXiv 2018.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject