Supernet Training for Federated Image Classification Under System Heterogeneity
Authors: Taehyeon Kim
Se-Young Yun
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated Learning
Image Classification
Supernet Training
System Heterogeneity

Abstract: Efficient deployment of deep neural networks across many devices and resource constraints, particularly on edge devices, is one of the most challenging problems in the presence of data-privacy preservation issues. Conventional approaches have evolved to either improve a single global model while keeping each local heterogeneous training data decentralized (i.e. data heterogeneity; Federated Learning (FL)) or to train an overarching network that supports diverse architectural settings to address heterogeneous systems equipped with different computational capabilities (i.e. system heterogeneity; Neural Architecture Search). However, few studies have considered both directions simultaneously. This paper proposes the federation of supernet training (FedSup) framework to consider both scenarios simultaneously, i.e., where clients send and receive a supernet that contains all possible architectures sampled from itself. The approach is inspired by observing that averaging parameters during model aggregation for FL is similar to weight-sharing in supernet training. Thus, the proposed FedSup framework combines a weight-sharing approach widely used for training single shot models with FL averaging (FedAvg). Furthermore, we develop an efficient algorithm (E-FedSup) by sending the sub-model to clients on the broadcast stage to reduce communication costs and training overhead, including several strategies to enhance supernet training in the FL environment. We verify the proposed approach with extensive empirical evaluations. The resulting framework also ensures data and model heterogeneity robustness on several standard benchmarks.

Under review as a conference paper at ICLR 2023
SUPERNET TRAINING FOR FEDERATED IMAGE CLASSI- FICATION UNDER SYSTEM HETEROGENEITY
Anonymous authors Paper under double-blind review
ABSTRACT
Efficient deployment of deep neural networks across many devices and resource constraints, particularly on edge devices, is one of the most challenging problems in the presence of data-privacy preservation issues. Conventional approaches have evolved to either improve a single global model while keeping each local heteroge- neous training data decentralized (i.e. data heterogeneity; Federated Learning (FL)) to train overarching network that supports diverse architectural settings address heterogeneous systems equipped with different computational capabilities (i.e. system heterogeneity; Neural Architecture Search). However, few studies have considered both directions simultancously. This paper proposes the federation of supernet training (FedSup) framework to consider both scenarios simultane- ously, i.c., where clients send and receive a supernet that contains all possible architectures sampled from itself. The approach inspired by observing that averaging parameters during model aggregation for FI. is similar to weight-sharing in supernet training. Thus, the proposed FedSup framework combines a weight- sharing approach widely used for training single shot models with FL averaging (FedAvg). Furthermore, we develop an efficient algorithm (E-FedSup) by sending the sub-model to clients on the broadcast stage to reduce communication costs and training overhead, including several strategies to enhance supernet training in the FL environment. We verify the proposed approach with extensive empirical evaluations. The resulting framework also ensures data and model heterogeneity robustness on several standard benchmarks.
INTRODUCTION
Deep neural networks (DNNs) have achieved remarkable empirical success in many machine learning applications. This has led increasing demand for training models using local data from mobile devices and the Internet of Things (IoT) because billions of local machines worldwide can bring more computational power and data quantities than central server system (Lim et al., 2020 El Sayed et al., 2018)). However, it remains somewhat arduous to deploy them efficienlly on diverse hardware platforms with significantly diverse specifications (e.g. latency, TPU) (Cai et al. 2019)
and subsequently train global model without sharing local data. Federated learning (FL) has become a popular paradigm for collaborative machine learning (Li et al., 2019 2018: Karimireddy et al., 2019; Mohri et al. 2019 Lin et al., 2020 Acar et al. 2021). Training the central server (e.g. service manager) in the FL framework requires that each client (c.g. mobile devices or the whole organization) individually updates its local model via their private data, with the global model subsequently updated using data from all local updates, and the process is repeated until convergence. Most notably, federated averaging (FedAvg) (McMahan al. (017)) uses averaging as its aggregation method over local learned models on clients, which helps avoid systematic privacy leakages (Voigt & Von dem Bussche 2017).
Despite the popularity of FL. developed models suffer from data heterogeneity as the locally generated data is not identically distributed. To tackle data heterogeneity, most FL studies have considered
new objective functions to aggregate of each model (Acar et al.) 2021 Wang et al. [2020] Yuan & Ma 2020; Li et al. 2021a). using auxiliary data in the centeral server (Lin et al. 2020 |020|Zhang et al.
2022), encoding the weight for an efficient communication stage (Wu et al. 2022 Hyeon- Woo et al.
2022 Xu et al. 202I), or recruiting helpful clients for more accurate global models (Li et al. 2019 Cho et al. 2020 Nishio & Yonetani 2019). Recently, there has also been tremendous interest in deploying the FL algorithms for real-world applications such as mobile devices and loT (Diao et al. et al. 2021; [Hycon- Woo et al., (022). However, significant issues remain regarding delivering compact models specialized for edge devices with widely diverse hardware platforms and efficiency constraints (Figure 1 (a)). It is notorious that the inference time of neural network

---

Under review as a conference paper at ICLR 2023
Big
Mediam
Small
Permonsication
Depbo
Server
IBT
Charth
Secver
Vaste
Superoct
Cherkt
Vanilla Network
Chaial
Ga Standard Fedented Learning
(b) Standard Supemet Training
10T
Doy
Sepernet
Chart
Supernet
FodSup
(d) Efficient FedSup
Figure 1: (a) Standard FI. framework, (b) supernet training in a standard datacenter optimization (i.e., centralized settings), (c) our proposed federation of superet training framework (FedSup), and (d) efficient FedSup algorithm (E-FedSup).
varies greatly depending on the specification of devices Yu et al. 2018). In this perspective, this can become significant bottleneck for aggregation rounds in synchronous FL training if the same sized
model is distributed to all clients without considering local resources (Li al. 2020).
In the early days, neural architecture search (NAS) studies suffer from system heterogeneity issues in deploying resource adaptive models to clients, but this challenge has been largely resolved by training single of shared weights from one-shot models (i.e. supernet) (Cai et al.. 2019 Yu et al.)
2020) (Figure I (b)). However, this approach has been rarely considered under data heterogeneity scenarios that can provoke the training instability, Recent works have studied the model heterogeneity
in FL by sampling or generating sub-networks (Mushtag et al. [202]] Diao et al. [021] Khodak 202T, IShamsian al. 202T), employing pruned models from global model (Horvath
et al. Luo et al.) 2021b However, these methods have limitations due to model scaling (e.g..
depth (flayers), width (fchannels), kernel size), training stability, and client personalization.
This paper presents novel framework to consider both scenarios, namely Federation of Supernet Truining (FedSup), i.e., sub-models nested supernet both data and model heterogeneity. Fed- Sup uses weight sharing in supernet training to forward supernets to each local client and ensembles the sub-model training sampled from itself at each client (Figure (c)). We manifest an Efficient FedSup (E-FedSup) which broadcasts sub-models to local clients in lieu of full supernets (Figure (d)). To evaluate both methods, we focus on improving the global accuracy (on servers, T.e., uni- versality) and the personalized accuracy (on-device tuned models, i.e., personalization). Our key contributions are summarized as follows:
We propose novel framework that simultaneously obtains a large number of sub-networks at once under data heterogeneity, and develop an efficient version that broadcasts sub-models for local training, dramatically reducing resource consumption during training, and hence the network bandwidth for clients and local training overheads.
To enhance the supernet training under federated scenarios, we propose new normalization technique named Parameterie Normalization (PN) which substitutes mean and variance batch statistics batch normalization. Our method protects the data privacy by not tracking running statistics of representations each hidden layer well reduces the discrepancies across shared normalization layers from different sub-networks.
We extend previous methods by analyzing the global accuracy and a personalized accuracy for each client where multiple dimensions (depth, width, kernel size) are dynamic; and demonstrate the superiority of our methods using accuracy with respect to FLOPS Pareto.
Experimental results confirm that FedSup and E-FedSup provide much richer representa- tions compared with current static training approaches several FL benchmark datasets, improving global and personalized client model accuracies.
2

---

Under review as a conference paper at ICLR 2023
Organization. The remainder of this paper is organized as follows. Section discusses relevant literature considering superett, model heterogeneity in FL, and personalized FL. the motivation for combining federated learning with supernet training and provides details regarding FedSup and E-FedSup. Section provides experimental results, and Section summarizes and concludes the paper.
discusses diseuses
2 RELATED WORK
Under the federated environment, architecture design costs are significantly labor-intensive and computationally prohibitive owing to the number of participating clients and local data quantities, even considering client network bandwidth requirements. We do not consider the works that assumed
the availability of proxy data in the server (Lin al. 2020 Zhang al., 2022).
Supernet. Supernets are dynamic neural networks that assembles all candidate architectures into weight-sharing network. where each architecture corresponds to one sub-network. This is an emerging research topic in deep learning, specifically NAS. Supernet training dramatically reduces the huge cost of searching, training, or fine-tuning each architecture individually whose child models can be directly deployed. However. despite its strength, supernet training remains somewhat challenging (Yu et al., 2018 Yu & Huang 2019). For the stable optimization, it requires many training techniques such as in-place knowledge distillation (Yu & Huang. 2019) to leverage soft predictions of the largest sub-network to supervise other sub-networks; modilied batch normalization to synchronize batch statistics for the child models Yuu et al. 2018: Yu & Huang, 2019); sampling strategies for child models from the supernet (Cai et al.] 2019 Wang et al. (021bb; and modified loss/gradient
functions Yu et al. 2020 Wang et al. 2021a).
Model Heterogeneity in FL. Model heterogeneity in FL., i.e., the problem of FL training different- size local models, has remained largely under-explored compared with statistical data heterogeneity. Recent studies have proposed generating sets of sub-models through a hypernetwork that outputs parameters for other neural networks (Shamsian et al., (02T), p. using a pruned model from global model (Bouacida et al.. 2020 Horvath et al., 2021: Luo et al. 2021b), and distilling the knowledge from local to global by using either extra proxy datasets or generators (Lin al. 2020 Afonin & Karimireddy 2022). However, pruning approaches are not truly cost-effective in terms of infer- ence time, and distillation-based methods require additional training overhead. Although using a hypernetwork (Shamsian et al. (02T) or sampling sub-model from the global model (Diao et al. 202I may avoid such issues, sub-model scale is limited to only a single direction, such as width or kernel size. Furthermore, such optimization is simple, hence the accuracy gap among sub-models should be bridged through advanced training techniques. Although Khodak et al. (202T) and Mushtag et al. (2021) apply continuous differentiable relaxation and gradient descent, the final outcomes significantly sensitive to hyperparameter choices.
Personalized FL. Machine learning-based personalization has become good candidate for retain- ing privacy and fairness as well recognizing particular local character. Consequently, personalized FL has been proposed to learn personalized local models, and these models have been evolving towards user clustering, designing new loss functions, meta-learning and model interpolation. Incor- porating meta-feature from all clients, local clients are clustered by measuring their data distribution and sharing separate models for each cluster without inter-cluster federation (Briggs et al. 2020 Mansour et al. 2020). Adding regularizer to the loss function can prevent local models from overfitting their local data (T Dinh et al., 2020: Li et al., 2021b), and bi-level optimization between clients and servers can be interpreted as model-agnostic meta-learning (MAML). This approach can obtain well-initialized shared global models that facilitate personalized generalization with relatively sparse one-ttning (Jiang et al. 2019: Oh et al., (02II). Lastly, decoupling the base and personalized layers in a network allows both layers to be trained by clients in addition to server base layers, creating unique models for each user (Oh et al. 2021 Chen & Chao 2021). However, few studies have considered personalized FL performance under the client system heterogeneity, i.e., clients with widely differing computational capabilities.
3 METHOD
3.1 PROBLEM SETTINGS: FEDERATED AVERAGING (FEDAVG)
The main FL goal is solve the optimization problem for distributed collection of heteroge- neous data: min (w) & min Ekes La (w)) where S is the set of total clients, Pt is the weight for client k. i.e., Pk and E Pk 1. The local objective for client is to minimize
3

---

Under review as a conference paper at ICLR 2023
Lx(w) = E ./.. ) JcD (Tx, Mk: w]] parameterized by w on the local data (æt, yk) from local data distribution Dr. FedAvg is the canonical algorithm for FL, based on local update, which learns a local model w (Eq. I with learning rate 7) and synchronizing wi with w' every E steps,
W w
( i mod E #
A
(I)
if mod E =
and global aggregation, which learns the global model w by averaging all wa with regard to the client & E 5 uniformly sampled at random where pk is defined as [Pal_ w = Exees Pk w TDT
Such FedAvg-base approaches facilitate the model generalization without accessing local data, but are unable to underpin various architectural settings for heterogeneous systems having different computational capabilities.
3.2 MOTIVATION FROM WEIGHT SHARING IN CENTRALIZED SUPERNET TRAINING
Let w be the weights of the supernet and Warch for sub-models. Then the problem for centralized supernet optimization can be expressed as
min Σ
Be (ED y; Werek)] where D = Dg
(2)
#Purch Cao
which can be reformulated as a double summation to preserve data privacy for FL frameworks,
min
Σ
De (ex (axx Yw: warch)
w
Warch Cuc
(3)
min
Σ Ps Σ
(a(wo))))
Architecture Design
Marck Cu
After exchanging the order of two summations, a new objec- tive (Bq.3) is obtained combination of training the supernet on each client's 's local data, termed as Federation of Supernet Training (FedSup) (Algorithm M.. We aim to design a new framework to combine the federated averaging scheme with weight sharing on mobile-friende friendly architectures. We attempt to apply several training techniques to bridge the gap among three dimensions (Figure 2}: (1) FL for data heterogeneity, (2) supernet training for model heterogeneity, and (3) mobilenet search space for architecture design. Most previous centralized supernet optimization studies heavily rely on improving train- ing stability by optimizing generalization for candidate networks sampled from the search space and adaptive sub-model selection at each iteration without considering data distribution. However, two new significant challenges arise in the supernet training in federated learning, referred to as federated supernet optimization, follows:
Datu
Nota Chattty
Heteregencity
Model Heteragebcity
IM
Figure Illustration for the prob- lem settings.
Sub-model alignment:
Data heterogeneity during training exacerbates distinct sub- aub-networks from interfering with one another. To address the training stability issue, it is required to align all of supernet's offspring models the same direction.
Client-aware sub-model sampling: Adaptive sub-model sampling strategy is required to draw more attention to client's individual data distributions and their resource budget, such as computational power and network bandwidth.
In this work, we emphasize the alignment property of the submodel to improve generalization ability and training stability in subsection 3.3 and explain the LLOPS-awand sampling strategy; child models are sampled by considering the chient's 's computational capacity in subsection 3.4 and subsection 3.5
3.3 TRAINING STRATEGIES FOR FEDSUP
Architecture Space. Search space details are presented by referring the previous NAS and FL approaches (Cai et al., 2019: Oh et al. 2021). Our network architecture comprises a stack with MobileNet VI blocks Howaard et al. 2017). and the detailed search space is summarized in Appendix. Arbitrary numbers of layers, channels, and kernel sizes can be sampled from the proposed network. Following previous settings YYu al. 2018 2020), lower-index layers in each network stage always kept. Both kernel size and channel numbers can be adjusted in a layer-wise wise manner.
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Generic Framework for FedSup and E-FedSup
INPUT Supernet w, the number of sampled child models M, weight update function UPDATE
1: Initialize SuperNet u7p
2: for f- O,..., 1 do
3: S SAMPLECLIENTS
4: for each client 5t in parallel do
5: wx s w
// Broadcast supernet w to client
6:
for - 0,
E do
7:
for m t. W archs..m weetl Lett archs.m
L,.... M do
8:
RANDOMSAMPLEMODELI
// Sub-modell selection
9:
OPTIMIZE( " archs.m
10:
end for lettl
11:
(erts 1ett Wik UUDATE(W
archu,.""
Get1 archy,
// Supernet optimization
end for
13: end for w1++1
EkEST p
// Aggregation
14: end for
Privacy
Heterogeneous & varianee
Sensitive query data
Statie
Slimmable
Slimmable
Scale and Shift
Update &
Track running
Scale and Shift
Instant batch statisties
Scale and Shift
Learnable parameter
Standardization
statistics
Standardization
Standardization
(a) Standard normalization
Static batch normalization
(c) Our parametric normalization
Figure 3: (a) Standard batch normalization, (b) static batch normalization Diao et al.
2021 and (c)
proposed parametric normalization.
3.3. PARAMETRIC NORMALIZATION (PN)
Although batch normalization (BN) is an ubiquitous technique for most recent deep learning ap- proaches, its running statistics can disturb learning for the sub-model alignment in supernet optimiza- tion because different lengths of representations can have heterogeneous statistics, and hence their moving average diverges for a shared BN layer YYu & Huang 2019 Yu et al.) 2018) (Figure (a)). Furthermore, is well-known that these BN statistics can violate the data privacy in FL (Li et al.) 2021c). To alleviate these issues, we develop new normalization technique, termed as parameteric normalization (PN), as follows:
Φ RMS( a) te
(approximated normalization with learnable parameter a)
(4)
y x + PNa,8,7(a)) (Affine: scale and shift)
(5)
where æ is an input batch data, RMS is the root mean square operator, 7, 3 are learnable parameters used in general batch normalization techniques, and a Iearnable parameter for approximating batch mean and variance. The key difference between our PN and previous normalization techniques is the existence of batch statisties parameters. In the previous works Yu al. 2018 Diao al. 2021), since the running statistics batch normalization layers can not be accumu lated during local training in FL owing the violence of data privacy as well different model size (Huang al. 2021), running statistics are not tracked. An adhoc solution employs static batch normalization (Diao al. 202T) for model heterogeneity in FL. with the running statistics updated as each test data sequentially queried for evaluation (Figure (b)). However, operational performance is highly dependent on such query data and can be easily degraded by its running statistics. In contrast, the proposed PN method simply uses RMS norm and learnable parameter rather than batch-wise mean and variance statistics, and thus does not require any running statistics. Hence more robust architectures can be trained towards batch data (Figure (c)). In nutshell, our PN methods not only eliminates privacy infringement concerns due to running statistics, but also enables slimmable normalization techniques robust towards query data.
5

---

Under review as a conference paper at ICLR 2023
3.3.2 IN-PLACE DISTILLATION
In our framework, the penalized risk for the local optimization of selected sub-models (line 9 in Algorithm D is calculated as
W
archs
arg min Lg (Warchy,) = arg min Warchi
[uw
+
L (Warchh))) La
(6)
Waroky
Standard FL loss
Sub modem alignmment loss
This generates sub-model alignment loss compared with standard FL optimization, which should be minimized by converging the representation divergence between the supernet and its child model, i.e., in-place distillation (Yu al. 2018):
(a yki w)
LE (Warh))))) - (t(w) = E log (IT; yk; Warchy)
where f()) is a neural network. From this perspective, a sub-model should be distilled during local training with soft labels predicted by the full model (biggest child). We set the temperature hyperparameter and the balancing hyperparameter between distillation and target loss (Hinton et al.) 2013) following Lin al. (2020).
3.4
E-FEDSUP FOR COMPUTATIONAL POWER AND NETWORK BANDWIDTH
While FedSup can solve system heterogeneity by distributing subnetworks in the deployment stage after training, this can cause computational bottlenecks for mobile or tiny AI during broadcast and local training. To further release this issue, we provide an efficient version of FedSup (E-FedSup) by optimizing alternative objective function instead of Eq. without employing in in-place distillation:
min Σ Ph Σ
Fk(Worch) min Σ P&FF (Warche)
where Warchy C
C
J:
FedSup
E-FedSup
Each local client receives a sub-model in the broadcast stage to achieve the efficient network band- width as a substitute of full supernet (line 5 in Algorithm [ For more details of E-FedSup imple- mentation, in Algorithm|] [ M is set to I and the with FLOPS-aware sampling function. Precisely, clients are clustered into groups with similar capabilities (i.e., call it "tiers") and each Warcha is randomly sampled until its computational budget (i.e., FLOPS) does not exceed the maximum FLOPS for the tier. A chief difference in the optimization is that FedSup samples new sub-model every iteration; whereas E-FedSup trains pre-fixed sub-model received during communication at local. Therefore, various sized sub-models delivered to different clients during every broadcast (i.c., Warch, * Warch, when i & in Eq. (8) As Figure (c) and (d) show, E-FedSup reduces communication costs by sending child models in the consideration of each local capacity, and also curtails training overhead because it trains the same child per iteration rather than several sub-models.
function is replaced
3.5 DETAILS FOR MODEL AGGREGATION AND CHILD SELECTION
Model A Aggregation. Typical implementations for FedSup and E-FedSup are based on the Algo- rithm The central server for FedSup uses FedAvg with [S'| number of local models updated with supernet optimization; whereas some clients during E-FedSup may have learned certain part of the weights Warch rather than all the parameters w. Non-updated parts of these models are filled with previously broadcast parameters and are regarded as a supernet for aggregation.
Sub-Model Selection for Deployment. A sub-network that fulfills accuracy and efficiency require- ments for the target hardware, e.g. latency and energy, is sampled by the central server for either model deployment after FedSup optimization or E-FedSup training. Although numerous child models can be sampled from the supernet, we divide clients into three tiers and candidate are explored trading off FLOPS and accuracy, denoted as "Big (B)", "Medium (M)", and "Small (S)". Neural networks with maximum and minimum resources determined by the architecture space are used for B and S cases, respectively (see Appendix); and neural networks with shallowest depth, largest width, and medium FLOPS for M cases (see elastic dimension results in Section Specialized model selection for a given deployment could be possible by assuming proxy data available in the server (Lin et al. 2020 Zhang al. 2022) select the pareto-frontier (subsection 4.2 but that is beyond the scope of the current study.

---

Under review as a conference paper at ICLR 2023
FodSup (Initial Acc.)
E-FodSup (Initial Acc.)
FodSup (Personalized Acc.)
E-FedSup (Porsonalized Acc.)
Width
Depth
Kerne
70-
81
Medium
Small
89
Mteium
Small
6i8
Medium
Small
(a) Width
(b) Depth
(c) Kemel
Figure 4: Initial and personalized accuracy on CIFAR-100 100 with 100 clients, 10, R - 0.1, and m = 0.5 by changing the slimmability with network width, depth, and kernel size. Blue lines indicate FedSup training, and the orange lines show E-FedSup performance.
EXPERIMENT
4.1 EXPERIMENTAL SETTINGS
Datasets. Four image classification benchmark datasets are employed: CIFAR-1O, CIFAR-
100 (Krizhevsky et al. 2009), Fashion-MNIST and pathMMNST for colon pathology classification, collection of standardized biomedical images (Yang et al. (222)).
Heterogeneous Distribution of Client Data. Two different kinds of heterogeneous data distribution settings are employed, following previous studies (McMahan et al. 2017 Oh al. [202T] Lin et al.) 2020). Datasets are divided into similarly sized shards by considering their label distribution. Since there is no overlapping data between shards, shard size is defined Nxe. where |D] is the data set size, NN is the total number of clients, and is the number of shards per user. The Dirichlet distribution is also used to create disjoint non-independent and identically distributed (non-i.i.d.) client training data (Yurochkin et al. [219 [Hsu et al., (209 Lin et al. 2020). Non-i.i.d. level is determined by the value of B. Where 3 = 100 simulates identical local data distributions, and clients with lower / are more likely to only hold examples from one class (randomly chosen).
Training Details. Each client has the same label distribution on local training and test data to help evaluate personalized accuracy. Following Oh al. (202T). FL environments are controlled using the following hyperparameters: client fraction ratio R, local epochs т, shards per user and Dirichlet concentration parameter 3. R is the ratio of participating clients to the total number of clients in each round and a small R is natural in the FL settings since the total number of clients is large. Linear warmup learning rate scheduler employed up to rounds, and subsequently scheduled using a cosine learning rate scheduler initialized with 0.1. Other settings not explicitly mentioned are followed by Oh et al. (202I). For the FedSup training, we use the number M of randomly sampled child model as equal to 3 and apply the in in-place distillation.
Evaluation. We evaluate the quality of the global model and local models, using the whole data set or following Wang et al, (2019) and Oh et al. 2202h, respectively. Personalization evaluation for each client is measured as follows:
Initial Accuracy. Transmitted global models are evaluated for each client having their own test dataset D!=
Personalized Accuracy. Local models are fine-tuned on their own training dataset D with Ty fine-tuning epochs; then evaluated on the client's 's own test dataset D
The values (X+y) in all tables indicate the mean of the accuracies across all clients, not across multiple seeds. Specifically, child models for the personalization are fine-tuned with five epochs on the local training data. We only update the head instead of training either full body part, following previous literature (Oh et al. 2021 Luo et al. 2021a). Further details regarding updated parts are provided in Appendix.
4.2
EVALUATION ON THE COMMON FEDERATED LEARNING SETTINGS
Elastic Dimensions. We firstly take closer look at applying the dynamic operations under federated settings towards three different dimensions: width, depth, and kernel size of convolutional operators. Conventional centralized learning methods (Tan & Le 2019) mostly scale ConvNets in one
7

---

Under review as a conference paper at ICLR 2023
Table 1: Initial and personalized accuracy on CIFARI00 under various FL settings with 100 clients
and r = 5; initial and personalized accuracy indicate the evaluated performance without fine-tuning and after five fine-tuning epochs for each client, respectively.
FL. Settings
5m50
Smi 10
FedSup
E. S-FeSSup
FedSup
E FedSSu
R
Initial
Personalized
Initial
Personalized
Initial
Personalized
Initial
Personalized
47.1926.01
58.55_0.37
57.90 57.96 29..49
25.24
71.21
27.60
69.81 16.77
1.0
15.1976.67
57.34-6.21
44. -4.34
34.81 56.80+6.44
70.56=6.68
68.01 34.72+6.69 +6.34
45.06
54..96=b.29
43. 23.09
53.98 53.98
34.01
70.21
33. 33.4147..0
68. 68.32
13.93
56.60_3.22
42.05
54.69 26.49
33.13
71.11
32.15
69. 69.1316.91
0.1
43.13
$5.87 25.36
41. 41.24_0.49
53.96
33.0447.78
70.03_6.ss
31.4228.77
69. 69.11
42.18
55.31
41.05.26.50
53.69
32.44
70.00=6.01
31.11
68.8646677
Table 2: Initial and personalized accuracy on CIFAR-100 with 100 clients, R = 0.1, and m 0.5.
Non N.D.L.LD
Algorithm
Local Oncal-Only
FedAvg 2017]
Ditio 2021b)
LG Fed/Avg 2020
Per: Fed/Fe
2020
FedSup
E: E.FedSup
Initial
36, 36.0317.58
33. 33.1227.61
30.01
38.551
43. 43.93
42.05 15 as
Personalized
27. 12
19. 19.61
11.10
39. 39.9225.00
44.21
56 20.60.5.20 22 54.69
Initial
21.58
20. 10
15.92.6.02 45.53:10045
21.21
33.113...01
32.1544
10
Porsonalized
60.33
64.51
58. 46.15
03.72+7.04
71 71.11+7.12
69.13 : us.
Table 3: Initial and personalized accuracy on CIFAR 100 between sBN (Dioo et al.
2021 and our PN.
FL Settings
50
x=10
FedSup
B FedSup
FedSup
E. FedSSp
R
N
Initial
Personalized
Initial
Personalized
Initial
Personalized
Initial
Personalized
sBN
17 14
58 14
46.18 18
08.04is.0 62
31.21 5 es
69. 69.42.2..0
29.77 2..22
69. 69.47:2.33
1.0
PN
47. 1m
58.55
45. 45.1911.1
57.9613.3
35. 24
71.21
34.6017.01
69.81
sBN
6
43.53 16
22
55.75 01
26.07
68.09
24.5627.8 a.
67.68
PN
43.93
56.60+5.2
42.05+5.30
54.69+6.45
33. 13
71.1147.12
32. 32.1540.44
69. 69.13+0.00
or other of these dimensions. As As Figure 4jshows, along any dimension, all child models are stably trained and can produce enhanced representations. FedSup has consistently better inital accuracies and personalized accuracies than E-FedSup. Dynamic depth exhibits little performance improvement compared with width or kernel size, which is desirable for further model scale optimization.
Compounding Dimensions. Table shows initial and personalized accuracies combining dimen- sions for architecture space. In most cases, FedSup has consistently less generalization eπror than E-FedSup in the global model (initial accuracy), and the gap between FedSup and E-FedSup after personalization keeps almost same when the five fine-tuning epochs are applied. Both FedSup and E-FedSup exhibit slight performance reduction as a model size gets smaller Table 2) By referring to the experimental settings in Oh et al. (202T), we also compare our methods with existing methods for universality and personality. Table 2 describes the initial and personalized accuracy of various algorithms. Other methods considered here are evaluated based
single model, whereas the proposed approach reports the performance of the largest network from supernet. Federated supernet optimization further improves model universality and personalization. Thus, the proposed approach verifies applying federated supernet optimization to future FL studies as an alternative to simple model averaging (FedAvg).
Parametric Normalization. Table compares static batch normalization (Diao al. 2021) and parametric normalization under federated supernet optimization. PN shows significantly better performance than sBN for s=10, which induces heterogeneous batch statistics; whereas the data distribution at 5=50, is closer to i.i.d,, and hence sBN has marginally better accuracy than PN in some cases. Since PN does not depend on query data as well as not contain any running statistics in its module, in contrast to sBN, there is little privacy concern. In this view, demonstrate that PN presents a more reasonable choice for federated supernet optimization.
We compare the accuracy FLOPS Pareto to find the set of solutions where improving one objective will degrade another in the multi multi-objective optimization (Eriksson et al. 2021). Here, we randomly sample 500 sub-models from the supernet and estimate their initial and
Pareto Frontier.
8

---

Under review as a conference paper at ICLR 2023
FedSup
E-FalSap
36.0
Neus Actracee
Moan of Acou DO0G
25.2
-4.
34.-
16
12
10
30
10
12
i
FIOPS
(LOPS(M) (c) 500 Acc.
FLOPS
(4) 101 Initial Ace.
(h) 1-10, Penvrralioed Ace.
(8) 500 Personaiiaal
Figure 5: Attributes (e.g., initial acc., personalized acc., and FLOPS) from trained supernet. Each dot represents a sub-model and 500 child models are sampled from the supernet.
personalized accuracies. As Figure 5) shows, larger models exhibit stronger performance for both methods. Even with the same FLOPS, the initial and personalized accuracy are different depending on the client's 's data distribution or sampled architecture. The performance degradation decreases almost linearly with decreasing FLOPS. FedSup has better Pareto frontier than E-FedSup for the initial and personalized accuracy.
FolAvy
HotcrFFL
We provide more investigations about collaboration with other methods such as FedProx (Li al., (018), in-place distil- lation, and label smoothing as well as the hidden representation changes (Kornblith et al., 2019) in the Appendix.
Others.
UedSop
U-VedSSp
Accuncy (55)
4.3 EVALUATION ON THE RESOURCE EFFICIENCY
Communication Cost Analysis. Figure deppict the communi-
cation cost required for the neural networks trained with FedAvg to reach 36% initial accuracy on CIFAR-TO0 = 100, R = 0.1, 50). The communication cost is paid until reaching the same accu- racy compared. The size of the model is about 1.96 MB, and thus the communication cost required until convergence is about 10 GB. Meanwhile, in order to achieve comparable levels of initial accu- racy, FedSup requires fewer epochs, making it efficient. In addition to that, E- E-FedSup is approximately four times more efficient than FedAvg by delivering sub-models rather than all of them.
Communication Cost (GB)
5,0
Figure 6: Communication costs with FedAvg, HeteroFL (2021).
FedSup, and E-FedSup.
0.40
0.35
Inference time (md
0.30
Inference Time. Figure 7 compares sub-model inference times for the (jORD.(Horvate et al.) (021)) and HeteroFL (Diao et al. 202T) model heterogeneity methods, measured on an NVIDIA 2080-Ti GPU. FedSup spawns much more efficient models in terms of local inference time. Since FjORD and HeteroFL are either unstructured pruning channel pruning-based methodologies, re- spectively, ours requires less processing time, benefitting from dynamic depth.
0.20
0.15
0.10
0.05
0.00
FORDD
NeterafL
RedSup
Figure Inference time per im- age for each sub-model.
CONCLUSION
This paper proposes the federation of supernet training (FedSup) branch of approaches under system heterogeneity motivated by the weigh-sharing utilized for the training of a supernet whereby contains all possible architectures sampled from itself. Our work engages in the solutions of federated learning for both data-heterogennitty and model--hetroogeeiityy Specifically, FedSup aggregates a large number of sub-networks with different capabilities into single one global model. We subsequently develop an efficient version of FedSup (E-FedSup) which reduces the model size transferred per round and local training overhead. We show that FedSup and E-FedSup have an excellent ability to generalize the personal data as well as the global data. Our methods yield strong results on standard benchmarks and medical dataset (Appendix) for federated scenarios; offering additional benefits by reducing network bandwidth and computation overheads on inference compared with conventional approaches. Despite the promise of FL, practical applications have been limited due to the large bandwidth requirement and computational capabilities for each local device. We believe that our work opens the door to deploying resource-ades aaduuree- service access to real-world applications with diverse system capabilities; while dramatically reducing energy consumption from training costs. This will considerably lower barriers to entry for developing dynamic FL models for DL practitioners and greatly impact the IoT industry.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough. and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference on Learning Representations, 2021.
Andrei Afonin and Sai Praneeth Karimireddy. Towards model agnostic federated learning using knowledge distillation. In International Conference on Learning Representations, 2022. URL https:
net htte:/oormmmiiiiieee m2 jv
Nader Bouacida, Jiahui Hou, Hui Zang, and Xin Liu. Adaptive federated dropout: Improving communication efficiency and generalization for federated learning. CoRR, abs/201 1. .04050, 2020.
Christopher Briggs, Zhong Fan, and Peter Andras. Federated learning with hierarchical clustering of local updates improve training on non-iid data. In 2020 International Joint Conference on Neural Networks (JJCNN) pp. 1-9. IEEE, 2020.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint 1812.v:812..0333 2018.
Han Cai, Chuang Gan, and Song Han. Once for all: Train one network and specialize it for efficient deployment. CoRR, abs/1 0999999 2019.
Hong-You You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations, 2021.
Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv preprint arXiv:2010.01243, 2020.
Enmao Diao, Jie Ding, and Vahid Tarokh. Hetero[fl]: Computation and communication efficient fed- erated learning for heterogeneoous clients. In International Conference Learning Representations, 2021. URL htttps://openreveee
net waw.iim----NNKPPAE.X
Hesham El-Sayed, Sharmi Sankar, Mukesh Prasad, Deepak Puthal, Akshansh Gupta, Manoranjan Mohanty, and Chin-Teng Lin. Edge of things: The big picture on the integration of edge, iot and the cloud in distributed computing environment. IEEE Access, 6:1706-1717, 2018. doi: 10.1 10./CEESCCS 2017
David Eriksson, Pierce I-Jen Chuang, Sam Daulton, Ahmed Aly, Arun Babu, Akshat Shrivastava, Peng Xia, Shicong Zhao, Ganesh Venkatesh, and Maximilian Balandat. Latency-aware neural architecture search with multi-objective bayesian optimization. arXiv preprint arXiv:2106.1 1890, 2021.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-leaming approach. Advances in Neural Information Processing Systems, 33, 2020.
Yuan Gao, Haoping Bai, Zequn Jie, Jiayi Ma, Kui Jia, and Wei Liu. Mtl-nas: Task- Task-agnostic neural architecture search towards general-purpose multi-task learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 11543-11552, 2020.
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
Geoffrey Hinton, Oriol Vinyals. Jeff Dean, al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7), 2015.
Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 34, 2021.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: /704.04861 2017.
10

---

Under review as a conference paper at ICLR 2023
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non- non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.0633:: 2019.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens Van Der Maaten, and Kilian Q Wein- berger. Multi-scale dense networks for resource efficient image classification. arXiv preprint arXiv: 09844, 2017.
Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, and Sanjeev Arora. Evaluating gradient inversion attacks and defenses in federated learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL net wtt.:eeeooooomoooooooooooeo/
https::/openzett
Nam Hyeon-Woo, Moon Ye-Bin, and Tae- Hyun Oh. Fedpara: Low-rank hadamard product for communication-efinn federated learning. In Interaational Conference on Learning Representa- tions, 2022. URL https:: http::::///ppeeeeeiiiiie net wnw.ee/eooouurtddaaaddrtera
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning, arXiv preprint (tti::/909./449000 2019.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for on-device federated learning. arXiv preprint arXiv:1910.0633 2019.
Jakob Nikolas Kather, Johannes Krisam, Pormpimol Charoentong, Tom Luedde, Esther Herpel, Cleo- Aron Weis, Timo Gaiser, Alexander Marx, Nektarios A Valous, Dyke Ferber, et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS medicine, 16(1):e 1002730. 2019.
Mikhail Khodak, Renbo Tu, Tian Li, Liam Li, Maria-Florina F Balcan, Virginia Smith, and Ameet Talwalkar. Federated hyperparameter tuning: Challenges, baselines, and connections to weight- sharing. Advances Neural Information Processing Systems,
2021.
Taehyeon Kim, Jongwoo Ko, Sangwook Cho, JinHwan Choi, and Se- Young Yun. FINE samples for learning with noisy labels. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL htttps://openreet http//ooummmideee 11ddddd00222222mm522
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference Machine Learning, pp. 3519- 3529. PMLR, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Jason Kuen, Xiangfei Kong, Zhe Lin, Gang Wang. Jianxiong Yin, Simon See, and Yap-Peng Tan. Stochastic downsampling for cost-adjustable inference and improved regularization in convolutional networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7929-7938, 2018.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10713-10722, 202 la
2018. Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith.
Federated optimization in heterogeneous networks, arXiv preprint arXiv:1812.066
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated Icarning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-6) 60, 2020.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357-6368, PMLR, 2021b.
Xiang Li, Kaixuan Huang, Wenhao Yang. Shusen Wang, and Zhihua Zhang, On the convergence of fedavg non-iid data. arXiv preprint arXiv: 0907.02 2019.

---

Under review as a conference paper at ICLR 2023
Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021c.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philippe Moreney. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXivv2001.01523, 2020.
Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang, Dusit Niyato, and Chunyan Miao. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys Tutorials, 22(3):2031-206 2020. doi: 10.1 10000///OOTTT.2022996666666
Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. Advances in neural information processing systems, 30, 2017.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. arXiv preprint arXiv:2006.07242, 2020.
Lanlan Liu and Jia Deng. Dynamic deep neural networks: Optimizing accuracy-efficienee trade-offs by selective execution. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34, 2021a.
Mi Luo, Fei Chen, Zhenguo Li, and Jiashi Feng. Architecture personalization in resource-constrained federated learning. 2021b. URL htttps://neurips/// wow.shopf www.shopf..ggthhu..... wit.ub. io/
NFFL-2021 2021/papersseeeeeeppppapaaaae /at.s/222//// 2ttp//22221222222111222222 Luo2021 supp.pdl
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for
personalization with applications to federated learning. arXiv preprint arXiv:2002. 10619, 2020.
Brendan McMMhan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-efficieett learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1282, 2017.
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. arXiv preprint 1rXiv:1922.001460 2019.
Erum Mushtaq, Chaoyang He, Jie Ding, and Salman Avestimehr. Spider: Searching personalized neural architecture for federated learning. arXiv preprint arXiv:2 /2. 13939, 2021.
Takayuki Nishio and Ryo Yonetani. Client selection for federated learning with heterogeneous resources in mobile edge. In ICC 2019-2019 IEEE International Conference on Communications (ICC), pp. 1-7. IEEE, 2019.
Jaehoon Oh, Sangmook Kim, and Se- Young Yun. Fedbabu: Towards enhanced representation for federated image classification. arXiv preprint arXiv:2106.06600 2021.
Ramakanth Pasunuru and Mohit Bansal. Continual and multi-task architecture search. arXiv preprint /rXiv:/906.005226, 2019.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. Advances in neural information processing systems, 30, 2017.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypemetworks. In International Conference on Machine Learning, pp. 9489-9502. PMLR, 2021.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv: 2019
12

---

Under review as a conference paper at ICLR 2023
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2822 2016.
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 153:1194--21000 2020.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105-61 14, PMLR, 2019.
Paul Voigt and Axel Von dem Bussche. The general data protection regulation (gdpr). A Practical
Guide, 1st Ed., Cham: Springer International Publishing, 10(3152666;:00-5555 2017.
Dilin Wang, Chengyue Gong, Meng Li, Qiang Liu, and Vikas Chandra. Alphanet: Improved
training of supernets with alpha-divergence. In International Conference on Machine Learning, pp. 10760-10771. PMLR, 2021a.
Dilin Wang, Meng Li, Chengyue Gong, and Vikas Chandra. Attentivenas: Improving neural architecture search via attentive sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6418 6418-6427, 2021b.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. arXiv preprint arXiv:2007.07481, 2020.
Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Fichner, Françoise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXiv preprint 1rX.v.1910.10252. 2019.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 409 420, 2018.
Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. Communicatioon-fficccnn federated learning via knowledge distillation. Nature communications, 13(1):1-8, 2022.
Zuxuan WU, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8817-8826, 2018.
Hang Xu, Kelly Kostopoulou, Aritra Dutta, Xin Li, Alexandros Ntoulas, and Panos Kalnis. Deepre- duce: A sparse-tensor communication framework for federated deep learning. Advances in Neural Information Processing Systems, 34:21150 1151211-1,,6,, 2021.
Brandon Yang, Gabriel Bender, Quoe V Le, and Jiquan Ngiam. Condconv: Conditionally parameter- ized convolutions for efficient inference. Advances in Neural Information Processing Systems, 32, 2019.
Jiancheng Yang. Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2: A large-scale lightweight benchmark for 2d and 3d biomedical image classification. arXiv preprint arXiv:2 /0. 14795, 2021.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 1803-1811, 2019.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas S. Huang. Slimmable neural networks. CoRR, abs/1812.00898.. 2018
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang. and Quoe Le. Bignas: Scaling up neural archi- tecture search with big single-stage models. In European Conference on Computer Vision, pp. 702-717. Springer, 2020.
13

---

Under review as a conference paper at ICLR 2023
Honglin Yuan and Tengyu Ma. Federated accelerated stochastic gradient descent. arXiv preprint SElv:2006.0895500 2020.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Trong Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks, arXiv preprint arXiv:1905. /2022, 2019.
Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. arXiv preprint arXiv:2203.09249 2022.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and patter recognition, pp. 8697-8710, 2018.
14

---

Under review as a conference paper at ICLR 2023
A OVERVIEW OF APPENDIX
In this supplementary material, we present additional details, results, and experiments that are not included the main paper due the space limit.
B ETHICS STATEMENT
To address potential concerns, describe the ethical aspect in respects privacy, security, infras- tructure level gap, and energy consumption.
Privacy and Security. Despite the promise of FL., owing to the presence of malicious users or the stragglers in the network, some workers may disturb the protocols and send arbitrary/adversarial messages that disturbs the generalization during FL. Recently, to tackle the system heterogeneity, some works allow the server to use proxy data or transmit encrypted data from local to server, but may infringe on privacy. FedSup is also able to have such potential risks during communication. However, because FedSup can enable the training of models under heterogeneous system without using any proxy dataset, our methods could be uses as general solution to personalize the model, having less risks of privacy and security under system heterogeneity. Under adversarial attacks, would be a nice direction to investigate the defense methods regarding the robustness against such ad adversarial risks.
Infrastructure Level Gap. In real-world applications, there is bandwidth issues between clients and the server. More precisely, because of some limited-service access to areas where communication is rarely possible. Sending a model of the same size can greatly affect the synchronize training of FL with such infrastructure level gap. Because our work is efficient in terms of communication cost, we can deploy the model resource-adaptively. In addition, it is possible to use the model adaptively enough within the local according to the model resource and situation.
Energy Consumption. Our methods are more efficient than other methods in the respect of energy consumption: (1) communication efficiency and (2) design costs. Firstly, if E-FedSup is used, the sub-model is transferred local as substitute for the full supernet. Therefore, noticeable energy- saving effects can be obtained. On the other side, since our methods can design various architectures rather than specialized neural networks, our approach reduces the cost of specialized deep learning deployment from O(N) to ((1) (Cai et al. 2201)). Even, our methods have less generalization errors than other FedAvg-variee variant methods while total communication costs are the same, so further energy-saving can happen in the respect of convergence speed.
Fed.vg-Varria Variant Methods
E-FedSup
FedSup & E-FedSup
Communication Cost
Design Cost
Reduction
Reduction
Number of Rounds
Number of Deployment Scenarios
(a) Communication Efficiency
(b) Design Efficiency
Figure Savings of energy consumption in the respects communication costs and design costs.
15

---

Under review as a conference paper at ICLR 2023
C LIMITATIONS AND FUTURE DIRECTIONS
In this sections, we describe the limitations of our work and future directions for further development.
Limitations. Although illustrating the superiority of our proposed methods state of the art, the bottleneck lies in the presence of arbitrary device unavailability or adversarial clients that disturb the training. We only consider vision-centric classification tasks on smaller datasets (CIFAR- 10, CIFAR- 100, Fashion-MNIST. PathMNIST). We do not investigate a large-scale datasets (namely ImageNet); FL. framework gets computationally more prohibitive the number of clients and local training iterations are increasing.
Future Directions. In future work, we aim to explore more efficient training strategics in the pres- ence of stragglers and adversarial users. Furthermore, improve the robustness of FedSup families in more resource-intensiin settings. We intend to investigate our methods on other applications such object-detection, semantic segmentation, and natural language processing models. Lastly, plan to explore why FedSup and E-FedSup has similar personalization accuracies while the global accuracy has slight gap between FedSup and E-FedSup.
D CONCEPTUAL COMPARISON TO PRIOR WORKS
Recently, numerous studies have been studied to address the problem of either data heterogeneity or model heterogeneity. As discussed in Section 2 literature can be categorized into several groups with FedSup and B-FedSup: data heterogeneity, model heterogeneity, and their hybrid. Table 4| systematically compares related methods in the respect to flexibility, data-privacy, efficiency on design, and efficiency on communication. More detailed explanations are described in Section To the best of our knowledge, our methods are the first method to satisfy the two conflicting factors: compounding model scales (depth, width, kernel size) and personalized models, by taking advantage of both categories.
Table 4: Comparison with related training methods: each method is grouped into three categories. In the first row, "Flexibilityy": need not be tied with specific architecture; "Data Privacy": keep the data privacy on each client; "Efficiency on Design": can design the architecture efficiently; "Efficiency on C:mmunication": can reduce the communication cost between clients and the server.
Category
Data Heterogeneity
Model Heterreeneity
Hyhrid
Method
FedAvg 2017
AFD 2020
FelDF 2020
OFA 2019
BigNAS 2020
FedSup
E-FedSup
Flexibility Data-Privccy
X
XOX
Efficieny on Design
X
X
Efficiency Communication
X
Dynamic Neural Network. Compared to static neural networks, dynamic neural networks can adapt their structures or parameters to input during inference considering the quality-cost trade- off (Han et al. 2021). To adaptively allocate computations on demand at inference, some works selectively activate model components (e.g., layers (Huang et al.] (017). channels (Lin et al. 2017 Sabour al. 2017)) )) controller or gating modules learned to dynamically choose which layers
of a deep network (Wu et al.) 2018 Liu & Deng 2018: Wang et al. 2018)); Kuen et al. (0118)
introduce stochastic downsampling points to adaptively reduce the feature map size. By extending the capabilities of well- knoln numan-designer neural networks like the MobileNet series (Howard let al. 20177; Sandler et al., 2018). Slimmable nets (Yu al.) [0018] Yu & Huang 2019) train itself by changing multiple width multipliers (for instance, 4 different global width mulupliers).
Benefits of Supernet. Many applications present the use cases of supernet for real-world scenarios. One of the most notable advantages is that they are able to allocate the user-customized network in consideration their capabilities on edge devices (e.g., smartphones, the internet of things) (Cai ett al., 2018). Next, the supernet seemingly produces better representation power than the static version of the network (Yang et al. 2019 Cai et al.) 2019)). In addition, supernet alleviates the issue of excessive energy consumption and CO2 emission caused by designing specialized DNNs for
16

---

Under review as a conference paper at ICLR 2023
every scenario (Strubell al. 2019: Cai et al. 2019). Lastly, supernet has superior transferability
across different datasets (Zoph et al. 2018) and tasks (Pasunuru Bansal 2019: Gao et al. (020).
All these advantages seem like double line that will work well in federated environment, to our best knowledge, but there are few studies applied in FL yet. Recently, Diao et al. (202T) show the possibility of coordinatively training local models by using weight-sharing concept while it limits the degree of flexibility (e.g., only width multiplier can adapt), analysis of model behavior, the examination for collection of training refinements, and the investigation towards personalization.
E IMPLEMENTATION DETAILS FOR SECTION 4
build our methods and reproduce all experimental results referring to other official repositories
E. ARCHITECTURAL SPACE
In this section, present the details of our search space. Our network architectures consist of stack with MobilenetVI blocks (MBConv) (Howard et al., 2017). The detailed search space is summarized in Table 5| For the depth dimension, our network has five stages (excluding the first convolutional layer (also called Stem)). Each stage has multiple choices of the number of layers, the number of channels and kernel size.
Table 5: MobileNerVI-baseee search space.
Stage
Operator
Resolution
#Channels
#Layers
Kernel Sizes
Conv
32x32
32
1
3
MBConv
16x16
32-64
1-1
3,5,7
2
MBConv
16x16
64-128
1-2
3,5,7
3
MBConv
8x8
128-256
1-2
3,5,7
MBConv
4x4
256-1024
1-2
3,5,7
E.2 M SAMPLED CHILD MODELS AND SANDWICH RULE.
At every local training iteration, the gradients aggregated from M sampled child models. If M 2 3, the smallest child and the biggest child are included where the gradients are clipped (i.c., sandwich rule Yu et al. 2018 2020 Through these aggregated gradients, supernet's weight is updated where the "smallest" child denotes the model having the thinnest width, shallowest depth, and smallest kernel size under the pre-defined architecture space.
E.3 EXPERIMENTAL. SETTINGS
Data Preprocessing. We use the same settings in Oh al. (202)) We apply normalization and simple data augmentation techniques (random crop and horizontal flip) on the training sets of all datasets, The size of the random crop set to 32 for all datasets referred to previous works (Oh et al.) 202
Liang et al. 2020 McMahan et al. 2017).
Dirichlet Distribution. To simulate a wide range of non-i.i.d.ness, we design representative heterogeneity settings based on widely used techniques (Yurochkin al. 2019). A dataset is partitioned by following Pe (ir I' that involves allocating Pkçe proportion of data examples for class client where I is the vector of ones.
17

---

Under review as a conference paper at ICLR 2023
CIFAR-10. CIFAR-II 10 (Krizhevsky al. 2009) is the popular classification benchmark dataset. CIFAR- 10 consists of 32 x 32 resolution images in 10 classes, with 6,000 images per class. We use 50,000 images for training and 10,000 images for testing.
CIFAR-100. CIFAR-100 (Krizhevsky et al. 2009) is the popular classification benchmark dataset. CIFAR- consists 32 32 resolution images in 100 classes, with 6,00 images per class. We use 50,000 images for training and 10,000 images for testing.
Fashion-MNIST is a dataset consisting of a 60,000 images for training and 10,000 images for testing. Each example is 28x28 grayscale image, associated with label from 10 classes. our work, an imput is rescaled into 32 x 32 resolution RGB images for data processing.
Fashion-MNIST.
PathMNIST. PathMNIST (Kather et al. 2019) is collection of 10 pre-processed medical open datasets. It is standardized to perform classification tasks on light weight 28 x 28 images, which requires background knowledge, while apply the image size 32 32. PathMNIST has classes and three subsets: training, validation, and test. Each has 89,996 data whose label distribution near balanced, but unbalanced, and we do not use the validation subset for training. FFigure showw several images from the training dataset.
Figure 9: PathMNIST Images.
18

---

Under review as a conference paper at ICLR 2023
Specification. We describe the detailed specification regarding 'Big', 'Medium', 'Small' models. Deservedly, other medium-sine models also are able to be sampled from the supernet while the trade-off between resourees and accuracies happens (Figure 5).
Table Specification for the child models sampled from the supernet. We report inference time in milliseconds, model size in million (M) units, and FLOPS in million (M) units of parameters.
Child Model
Big (B)
Medium (M)
Small (S)
Inference Time
0.37 (ms)
0.20 (ms)
0.06 (ms)
Model Size
1.96 (M)
1.47 (M)
0.78 (M)
FLOPS
13.36 (M)
7.51 (M)
4.00 (M)
F ADDITIONAL EXPERIMENTAL RESULTS
F.1 PERSONALIZAATION AND CKA SIMILARITIES
Body
Head
Full
FedSup
E-FedSup
FedSup vs
E-FedSup
0.8
FedSup
08
10
50
07
20
20
04
02
RedS
E-FedSup
Layers E-FedSup
25
60
Layers E-FedSup
20
40
00
Updates porsonalization
(b) sK11
C.A,,,
Figure 10: (a) Personalization accuracy of FedSup and E-FedSup on CIFAR-100 according to the fine-tuned part by referring to Oh al. (202T) (other parts freezed); (b), (c): Centered Kernel Alignments (CKA) similarities of two different global models trained with FedSup and E-FedSup (Kornblith al. 2019).
Personalization. Referring to recent personalized FL experimental settings (Oh et al. 2021 we compare the performance according to the fine-tuned part (Figure (a)). Child models are fine-tuned with five epochs based on the local training data. As mentioned in literature (Oh et al. 2021| Luo al. 2021a), is shown that updating only head has slightly better performance than the others including local-only training TTble 2). In the main paper, we thus updates only the head for the personalization unless otherwise mentioned.
CKA Similarities (Kornblith et al. 2019). We vividly compare how the representations of neural networks are changed through the FedSup and E-FedSup. To be specific, Centered Kernel Align- ment (CKA) is leveraged to analyze the features learned by two architectures trained with FedSup and E-FedSup under different heterogeneous settings, given the same input testing samples (Figure (b) and (c)). Regardless of the degree of heterogencity, CKA visualizations show that the representations of two neural networks trained with FedSup and E-FedSup seem similar during the propagation.
F.2 GLOBAL ACCURACIES
Unlike the main section, we evaluate a global accuracy of each server model with original test dataset on CIFAR-I0 dataset [Tall 7 Table 8)
F.3 ABLATION STUDIES: MOMENTUM AND LABEL SMOOTHING
Momentum. Table 9 describes the initial and personalzied accuracy according the momentum.
The momentum is not applied during the fine-tuning of personalization. In most cases, appropriate momentum improves the performance.
Label Smoothing (LS) SZegedyy et al. 2016). Table 10 describes the initial and personalized
accuracy according to the LSS LS is popularly used in the existing weight-sharing methods (Cai et al. 2019 Yu et al., (020: Wang et al. 2021b), but in environment, it rather degrades both initial and personalized performance.
19

---

Under review as a conference paper at ICLR 2023
Table Performance of FedSup on CIFAR-I0 test dataset. (last accuracy / best accuracy) written in order (R = 0.1, N = 100).
Dirichlet
Shard
B = 0.01
[ = 1.0
s=2
s=10
27.31 29.04
72.34 / 73.04
/ 58.70
74.38
0.1
34.98 / 35.05
74.51 / 74.88
60.22 / 61.58
75.51 76.24
0.5
35.12 35.72
75.22/ 76.10
61.42 / 62.37
75.13 76.90
0.0
44.21
/45.31
80.13/ 00.13/80.00 80.80
69.30 69.39
81.01 88,66
5
0.1
47.15 / 47.77
82.22/ / 83.02
69.62 / 70.61
82.15 82.92
0.5
51.81 / 52.13
82.60 83.18
70.20/ 01.05
82.14 82.98
Table 8: Performance of E-FedSup on CIFAR-I0 test dataset. (last accuracy best accuracy) written in order (R = 0.1, N = 100).
Dirichlet
Shard
8 = 0.01
[ = 1.0
s=2
s=10
25.49 27.02
72.27 72.62
60.67 60.70
/ 73.63
1
0.1
28.56 / 30.87
72.55 / 72.56
60.72 / 60.93
73,28 74.01
0.5
33.34 33.34
73.32/ 74.38
60.16/ 61.25
73.21 / 74.87
0.0
40.72
41.00
80.06 / 80.52
64.64 / 67.10
79.38 80.04
5
0.1
42.43
/ 44.83
79.83 / 80.87
65.54/ 67.96
80.90 / 80.92
0.5
46.36
46.48
80.39 / 82.14
65.70 68.49
80.72 80.98
F.4 EXPERIMENTS ON MEDICAL DATASET.
Table I1 shows that FedSup and E-FedSup work fairly well on the PathMNIST dataset and have the similar tendency shown in Table I
20

---

Under review as a conference paper at ICLR 2023
Table 9: Initial and personalized accuracy of FedSup and E-FedSup CIFAR-100 according the change of the momentum magnitude. The fine-tuning epochs is 5, R is 0.1, N is 100, and s is 10.
Settings
M
Alg.
Personalized
Initial
Personalized
Personalized
0.0
30. 117.98
68, 09
30. 28 73
67 .5888...000 6.....
29.7347.30
66. 66.7997.00
HedSup
31.04 $1.0647
69.98_..44
30.98 6s
68 54 50
30.35
67.61
0.5
33. 133.33...0
71.11
33.04
70.0346.8
32.44
70.00
29.91 29.91 17.14
66.94
29.621 15
66. 15
29. 11..80
65.8617.02
E-FedSup
0.1
10.399:: 26.88
68, 74
30. 30.06
G7 83
29. 72.004
66, 70...2...2
19.44
69.13
41.4224.77
69.1 69.1116.61
31.11ix.88
68.8626.77
Table 10: Initial and personalized accuracy of FedSup and E-FedSup CIFAR-100 with and without label smoothing. The fine-tuning epochs is 5, R is 0.1, N is 100, and s is 10.
Architecture Size
M
Architecture
Initial
Personalized
Initial
Personalized
Initial
Personalized
FodSup
0.0
33. 13
71.1147.12
33 0447.78
70. 20.58
32 42.444..99
70.00- 70.000+..0 +6.91
31.11 80
89.8617.54
30.70
26.74
30.83
16.
E. FedSSp
0.1
0.0
30.63.0.0
32.15 44
68.7746.33
69. 13...6
31.42 31.42
30. 30.15
67.41
67.41
69. 69.11
16.61
26.45
30.04
31.1114.85
66.97 #6.78
68. 68.86..
Table 11: Initial and personalized accuracy with static batch normalization on PathMNIST Yang et al. 2021 under various FL settings with 100 clients. We implement data heterogeneity through
Dirichlet distribution (3) Yurrochki et al. 2019; Hsu et al. 2019 Lin et al. 2020). FedAvg algorithm has 2-3% lower initial and personalzied acc. average than E-FedSup.
Fl. Settings
100.0
8 1.0
FodSup
E-FedSup
FodSup
E-FodSup
R
A
Initial
Personaized
Initial
Personalized
Initial
Personalized
Initial
Personalized
B
75.0224.05 74.67
73.04
73.5614.7
71.70_e.0
79.67 79.6716.34
70.33
17.1716.07
1.0
71.33
74.33
71.30+4.47
73. 24.69
70.19
78.63
69.40 45
78.18 27.38
74.03
73.1214.44
70.6616.50
70.0015.00
68.59_a.17
77.6017.17
66.95
76.38
74.76-4.22
44.38:4.20
73.91
73.2244.95
70.07 =0.07_
79.3017.20
69.10
26.74
0.1
M
73.97
73.48
74.0814.91
73.2614.54
69.46_9.23
78.8016.3
69. 13.1
78.7616.24
73. 23.23
71.79:4.us
us
73.88
72.97
68.05=a.77
47.3747.52
67. 67.27 wr
76.99 27.38
Table 12: Initial and personalized accuracy with parametric normalization on PathMNIST Yang et al. 202T) under various FL settings with 100 clients. We implement data heterogeneity through Dirichlet
distribution (3) (Yurochkin et al. 2019 Hsu et al. 2019 Lin et al. 2020). FedAvg algorithm has 3-4% lower initial and personalzied acc. on average than E- FedSup.
FL Settings
= 100.0
1.0
FodSup
E-FedSup
FodSup
E-FodSup
Initial
Personalized
Initial
Personalized
Initial
Personalized
Initial
Personalized
75.38
75.08+4.2 4.22
74.59
24.38
79.93 72.76-s.a5
70.95 70.98 79.78 +6.31
1.0
M
75. 75.05-44.24
74.89
74.30
73.69
71.17
78.54446.00
70.01
78.60 78.60
73.97
73
71.4625.75
70.9615.15
68.82_8.91
77.1247.77
69.64g4.8 s3
47.74
74.57
74.63 4a
74.51
73.6745.16
71.12_8.08
79.11
g4
79.02 79.02=6.00
M
73.86 73.4944.38
73.7146.3n
73.5345.12
70.38-9.00
78.9940.38
70.07 +..60
78.60 +6.54
73.27 S
73.56+5.11
73.38
72.73-4..0
69.87
=8.22
78.53477.0
68. 08.26gs.70
78.02 02.022...00
21

---

Under review as a conference paper at ICLR 2023
Table 13: Experiments with static batch normalization: Initial and personalized accuracy on Cl- FAR under various FL settings with 100 clients. The initial and personalized accuracy indicate the evaluated performance without fine-tuning and after five fine-tuning epochs for each client, respectively.
FL. Settings
50
s=10
FedSup
E-FedSup
FedSup
E-FedSup
R
Initial
Personalized
Initial
Personalized
Initial
Personalized
Initial
Personalized
B
42. 83 g5..os
$5.03
42. 46 26
55.9516.03
25. 25.9626.47
65.75
26. 33 16.33
66.44
41.3925.30 20.33
4.00
55.91
26. 26.04
65.59_6.00
66.50 26.50
01
39. 19.17744777
58.17=4.77
39.78-6.29
54.35+6.88
25.0646.94
64.81-6.12
25. 26.2000.00
64.53+0.52
47.08 M
58.15_6.14
58.0115.62 18.18
31.2415.60
69.42.6.co
29.77 a2
69.47
5
44.34
57.01
11.55
57.25
28. 11.111
69.37
30.22
69.38 18.00
40.3325.00
$2.78_5.00
40.22_5.28
53.24g5.43
25.01
66.4916.36
26.41.5..7
66.30
38.04 25.30
53.55-4.81
59.37=25400
54.88
22.43
65. 12.1.25..5
22. 22.42.15.52
64.75-16.3
M
38.09
53.31 266
39.33
54.81
22.40.5.23
64.95
22. 43..00
64.7346.
52.29_1..0
37. 37.3125/28
23.08
21.17
61.18
21.32 21.32.35.5
61.29116.6t
0.1
B
43.83.+0.20
56.51-s.15
43.58 43.53-4.22
66.70+6.01
26.07
08.09
24.6647..36
67. 67.08
M
12.21
55. 55.15
12. 12.24=2.72
55.38 25.1
21. 21..9
67 67.93
24.76
67.87 16.50
37.94
15.15
52.15
27.19
52.3315.39
20.27 16.98
64.41g6.30
20.28.16.71
64.35
22

---

Under review as a conference paper at ICLR 2023
F.5 LEARNING CURVE OF GLOBAL ACCURACY
We visualize the learning curves of the networks trained with FedSup and E-FedSup (Figure 11 As Figure 11 shows, FedSup has slightly better performance than E-FedSSp. Here, the cosine learning rate scheduler is used, and the detailed explanations are noted in Section 4
FedSuup E-FedSup
40
30
Accuracy
10
200
300
500
Communication Round
Figure 11: Learning curve of the networks trained with FedSup and E-FedSup. Both networks trained with 50, r = 5, = 0.1, 100.
F.6 PATHMNIST RESULTS OF FEDAVG
As mentioned in Table FedSup and E-FedSup works better than FedAvg algorithm. Most
performances in Table 14 are lower than the values in Table 14
Table 14: FedAvg performance on PathMNIST (N 100, 5).
100.0
1.0
R
Initial
Personalized
Initial
Personalized
1.0
0.0
72.21
72.09
69.95
77.00
0.1
0.0
71.15
72.12+4.69
67.87 46.78
76.77
F.7 FASHION-MNIST RESULTS
As Table 15 shows, FedSup and E--eedSup fairly works well on Fashion-MNIST dataset.
Table 15: Initial and personalized accuracies on Fashion-MNIST with 100 clients, R = 0.1, and
0.5.
Non-LLDD,
Algorithm
FodAvg 2017
Ditto
2021b
LG-FodAvg
3020
Por-FodAvg 2020
FodSup
E-FodSup
Initial Personalized
81.12+6.99
68.9 08.93+7..25
83.37+5.80
35.79
91 62
89 09.42+0.84
90.30
82.17
90.6445.
92.35 6.07
94.4715.71
94.0116.19
Initial
75.4316.57
62 17.7
81.45
79.96
86 86.23....1
83 03.2011..e0
Personalized
91.88g5
88.72 88.724...7 79
77.33.... 5a
85.01 1 .54
48.4844
93.3111.77
F.8
INPLACE DISTILLATION: REPRESENTATION DIVERGENCE
Table 16 describes the initial and personalzied accuracy according to the inplace distillation. The inplaceddistillation is not applied during the fine-tuning of personalization. In most cases, applying
23

---

Under review as a conference paper at ICLR 2023
inplace distillation improves the performance. Namely, without any additional models, it can supervise a sub-uuuuooooo uub-model'ss 8s representation aligning into the same direction (i.e., representation alignment; a concept from Kim et al. (Kim al. (021)).
Table 16: Initial and personalized accuracy of FedSup on CIFAR-100 with and without inplace distillation. The fine-tuning epochs is 5, R is 0.1, N is 100, and is 10.
Architecture Size
Architecture
In-Distll
Initial
Personalized
Initial
Personalized
Initial
Personalized
FodSup
True
33.13+8.01
71.11
33. 04 33.04-7.78
70.03
32 74.444+.00
70.00+6.0
False
32.8847.90
70.62
32. 33..0m
69.5527.07
30.76
68. 28.4447.51
F.9 FEDPROX: WEIGHT DIVERGENCE
Table 17 describes the initial and personalized accuracy according to the FedProx. The FedProx is not applied during the fine-tuning of personalization. most cases, there remains little changes in performance after applying FedProx. Here, we use the value of hyperparameter À in FedProx as 0.001.
Table 17: Initial and personalized accuracy of FedSup on CIFAR-100 with and without FedProx. The fine-tuning epochs is 5, R is 0.1, N is 100, and is 10.
Architecture Size
B
Architecture
FedProx
Initial
Personalized
Initial
Personalized
Tritial
Personalized
X
33 134
71. 71.11+7.12
33.04+7.re
70.03
32. 44 22.4444..0
70.00
FedSup
22.93
70.01
33.01
7.
32. 22.02
68.38 27
F.10 TRAINING TIME ANALYSIS ON SYNCHRONIZED TRAINING SETTINGS
Our methods are much more efficient terms of time than the synchronous training of FedAvg- Variant methods. Consider example for real-world applications. Since IoT, Edge Device, and Cloud Server have different resource performance, the time it takes for local training is different for each machine. We assume the local training time for every round in Tabll 18 If you need to train with FedAvg- Variant Model, the time it takes to synchronize every round is 30 (sec) + network bandwidth time. On the other hand, in the case of B-FedSup. the model is distributed in consideration of the resource, S for IoT, M for Edge Device, and B for Cloud Server, FL can be implemented so that 10 (sec) network bandwidth time required. FedSup can also be implemented much more effectively than FedAvg-vare variian methods if sub-models are selected well in local training.
Table 18: Assuming that the local training time of the Big model in the IoTT device is 30 seconds, the training time in different machines of different models assumed based on this.
B
ToT
30(sec)
20(sec)
10(sec)
Edge Device
20(scc)
10(sec)
6(sec)
Cloud Server
10(sec)
5(sec)
3(sec)
F.11 NUMBER OF SAMPLED ARCHITECTURES FOR TRAINING IN FEDSUP
We study the number of sampled architectures M per training iterations. It important because larger leads to more training time. We train the models with equal to 1,2,3, or where the sandwich rule is not applied when n <
24

---

Under review as a conference paper at ICLR 2023
Table 19: Performance of FedSup on CIFAR-10 test dataset with supernet having dynamic operations
on depth, kernel, and width. (accuracy on Dirichlet distribution having 3 0.01 / accuracy Dirichlet distribution having 3 = 1.0) is written in order (R 0.1, N 100, r = 5, m = 0.5).
W/ Sandwich Rule
47.90 80.01
48.53 80.76
W/O Sandwich Rule 4 79.22
46.99 79.15
46.11 /
47.16 79.91
25


Reviews:
[{"review_id": "c8PO3WyIO1", "reviewer": "Reviewer_Zkr8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper empirically studies the federated learning setting on super-net training.\nThis paper considers both directions simultaneously to overcome the data privacy issue and data heterogeneous issue in parallel.", "summary_of_the_review": "This paper studies a well-motivated interesting problem, paper is well-written and easy to follow. \n\nThis paper mainly focused on the engineering part of federated learning, however for papers that are working on federated learning, some theoretical analysis on either convergence or generalization is usually expected. \n\nThe proposed complexity-based sampling method might exaggerate the data heterogeneously issue and hurt model performance. \n\nThe authors are also expected to compare with centralized training to demonstrate the trade-off between efficiency and accuracy.\n", "strength_and_weaknesses": "Pros:\n\n1. The problem is well-motivated. Existing methods either \"train a single global model but keeping each local heterogeneous training data decentralized\" or \"train an overarching network that supports diverse architectural settings to address heterogeneous systems equipped with different computational capabilities\". This paper considers both directions simultaneously to overcome the data privacy issue and data heterogeneous issue in parallel.\n\n2. The paper is well written. The challenges of super-net training are well listed and solutions are provided in Section 3.\n\n\n\nCons:\n\n1. This paper is mainly focusing on the empirical part of federated learning and supernet training. However, it would be great if the authors could provide theoretical evidence on why \"parametric normalization\"  and \"in-place distillation\" could help alleviate the data heterogeneity issue from either optimization (e.g., convergence analysis) or generalization (e.g., the generalization analysis in [1]). \n\n\n[1] Personalized Federated Learning: A Meta-Learning Approach https://arxiv.org/abs/2002.07948\n\n2. Although sampling based on FLOPs is interesting, I am having doubts about whether this will exaggerate the data heterogeneity issue. For example, lower complexity models are more likely to be trained on devices with less computation ability, and the data will be loss similar on these devices.   \n\n3. For experiments, I think the authors also need to compare with single machine supernet training, i.e., all heterogeneous data are gathered onto a single device and used for training. This experiment could be used as a baseline to show the trade-off between efficiency and accuracy.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MIeOFBCZ6d", "reviewer": "Reviewer_zfru", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies supernet training in the federated setting. Compared to standard supernet training algorithms, various twists have been made to accommodate the constraints in federated learning systems. The experiments have verified the performance of the proposed method. ", "summary_of_the_review": "The topic is interesting, but the novelty is somewhat limited, and overall the paper should have more insightful discussions on both the technical contributions and empirical results.", "strength_and_weaknesses": "Strength: \n- Supernet training in federated learning is an interesting topic. \n- This work can potentially have practical value as it explains the detailed steps of the method. \n\nWeaknesses:\n- The paper describes a lot of the details in the approach. The high level picture is somewhat missing, so it is hard to identify a core novel contribution in this work. It seems the contribution is mainly in the detailed tuning of the centralized supernet training algorithm to the federated setting. I think such a work can be useful, but not very interesting from a scientific point of view. \n- It would be nice if non-obvious findings, both in the algorithm design and empirical results, can be highlighted and discussed in the paper. \n- It is not clear what Dirichlet parameter $\\beta$ is used to obtain the results shown in the main paper. \n- I wonder why the personalized model gives much better performance than the initial model, and why the initial model's accuracy is so low. Is there an explanation for this?\n- There are only a few plots that compares the proposed FedSup algorithm with baselines. To conclude that FedSup performs uniformly better than baseline algorithms, comparisons with baselines need to be made with multiple datasets. I wonder why the appendix does not include such comparisons for the other datasets. \n- What does O and X mean in Table 4?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C_IIeu62unH", "reviewer": "Reviewer_jbFv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "While federated learning has become a popular paradigm for collaborative machine learning, it suffers from data heterogeneity and system heterogeneity. This work proposes FedSup, a federation framework of supernet training to address both heterogeneities. The key idea is to directly train the subnetworks on the device and introduce parametric normalization and self-distillation to improve the training. Experiments show that FedSup improves global and personalized client model accuracies with better communication efficiency.", "summary_of_the_review": "This paper proposes an interesting question on training the supernet of neural architecture search in the context of federated learning. However,  In general, I think this paper is marginally above the acceptance threshold.", "strength_and_weaknesses": "Strength:\n+ this work provides an interesting analysis of similarities and differences between federated learning and supernet training in NAS.\n+ extensive experiments evaluate both FedSup and E-FedSup performance compared to conventional approaches (FedAvg etc).\n\nWeakness:\n- the analysis of the compounding effect in E-FedSup is weak. In E-FedSup, the system heterogeneity and data heterogeneity are entangled since sub-architecture sampling is biased. It would be better if this work could provide more experiments on the generalizability of the trained supernetworks under E-FedAvg.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject