DepthFL : Depthwise Federated Learning for Heterogeneous Clients
Authors: Minjae Kim
Sangyoon Yu
Suhyun Kim
Soo-Mook Moon
Venue: ICLR 2023 poster (2023)
Keywords: Federated Learning
Heterogeneity

Abstract: Federated learning is for training a global model without collecting private local data from clients. As they repeatedly need to upload locally-updated weights or gradients instead, clients require both computation and communication resources enough to participate in learning, but in reality their resources are heterogeneous. To enable resource-constrained clients to train smaller local models, width scaling techniques have been used, which reduces the channels of a global model. Unfortunately, width scaling suffers from heterogeneity of local models when averaging them, leading to a lower accuracy than when simply excluding resource-constrained clients from training. This paper proposes a new approach based on depth scaling called DepthFL. DepthFL defines local models of different depths by pruning the deepest layers off the global model, and allocates them to clients depending on their available resources. Since many clients do not have enough resources to train deep local models, this would make deep layers partially-trained with insufficient data, unlike shallow layers that are fully trained. DepthFL alleviates this problem by mutual self-distillation of knowledge among the classifiers of various depths within a local model. Our experiments show that depth-scaled local models build a global model better than width-scaled ones, and that self-distillation is highly effective in training data-insufficient deep layers.

Published as a conference paper at ICLR 2023
DEPTHFL: DEPTHWISE FEDERATED LEARNING FOR HETEROGENEOUS CLIENTS
Minjac Kim
Sangyoon Yu
Seoul National University mjkimêsnu, ai.kmm@nu.ac.kt ac.kr
Seoul National University sangyoonyuu snu ac. kr
Suhyun Kim
Soo-Mook Moon'
Korea Institute of Science and Technology dr suhyun, kime com
Seoul National University smoonesnu, ac kr
ABSTRACT
Federated learning is for training a global model without collecting private local data from clients. As they repeatedly need to upload locally-update weights gradients instead, clients require both computation and communication resources enough to participate in learning, but in reality their resources are heterogeneous. To enable resturce-constrained clients to train smaller local models, width scaling techniques have been used, which prunes the channels of global model. Unfor- tunately, width scaling suffers from parameter mismatches of channels when ag- gregating them, leading to lower accuracy than when simply excluding resource- constrained clients from training. This paper proposes a new approach based on depth scaling called DepthFL to solve this issue. DepthFL defines local models of different depths by pruning the deepest layers off the global model, and al- their resources. Since many clients do not
locates them clients depending have enough resources to train deep local models, this would make deep layers partially-trained with insufficient data, unlike shallow layers that fully trained. DepthFL. alleviates this problem by mutual self self-distillation of knowledge among the classifiers of various depths within local model. Our experiments show that depth-scaled local models build a global model better than width-scaled ones, and that self-distillation is highly effective in training data-insufficient deep layers.
INTRODUCTION
Federated learning is type of distributed learning. It trains shared global model by aggregating locally-updated model parameters without direct access to the data held by clients. It particularly suitable for training model with on-device private data, such next word prediction on-device item ranking (Bonawitz et al., 2019). Generally, federated learning demands client devices to have enough computing power train a deep model as well enough communication resources to exchange the model parameters with the server. However, the computation and communication ca- pability of each client is quite diverse, often changing dynamically depending on its current loads, which can make those clients with the smallest resources become bottleneck for federated learn- ing. To handle this issue, would be appropriate for clients have different-sized local model depending their available resources. However, it is unclear how we can create local models of different sizes, without affecting the convergence of the global model and its performance. There are various methods to prune single global model to create heterogeneous local models, such as HeteroFL (Diao et al. 2021), FjORD (Horváth et al.] 202T), and Split-Mix (Hong et al.) 2022). They create local model as subset of the global model by pruning channels, that width- width-based scaling. HeteroFL was a cornerstone research that could make different local models by dividing global model based on width, yet still producing global model successfully. However, observed some issues of width scaling. We evaluated HeteroFl. compared to exclusive federated learning, which
co-coorespooding authors

---

Published as a conference paper at ICLR 2023
simply excludes those clients who do not have enough resources train a given global model from training (see Section 4.2). The result shows that the global model of HeteroFL achieves a tangibly lower accuracy than the models of exclusive learning, due to parameter mismatch of channels when they are aggregated. In this paper, we propose a different approach of making local models, called DepthFl.. DepthFL. divides a global model based on depth rather than width. We construct global model that has several classifiers of different depths. Then, we prune the highest-level layers of the global model to create local models with different depths, thus with a different number of classifiers. We found that this depth-based scaling shows a better performance than exclusive learning in most cases, unlike in HeteroFL, since training a local model can directly supervise its sub-classifiers as well as its output classifier, obviating parameter mismatch of sub-classifier when aggregated. This means that depth scaling allows resource-constrained clients to participate and contribute to learn- ing, although there is a small overhead of separate classifiers. We analyzed the root cause of this difference between depth scaling and width scaling.
There is one issue in depth scaling, though. There are only few clients whose local models include deep classifiers, while many clients have shallow classifiers in their local model. This would make deep classifiers be partially-train trained only with limited amount of data, their accuracy might be inferior to uully-trained shallow classifiers. That is, resource-conse constrained clients cannot train deep classifiers, so deep classifiers cannot be general enough cover the unseen data of those clients. To moderate this issue, we make the deep classifiers of a local model learn from its shallow classifiers by knowledge distillation. This is similar to self distillation Zhang et al. 2019). except that the direction of distillation is opposite. Actually, we make the classifiers collaborate with each other as in deep mutual learning (Zhang et al., 2018). So, client not only trains the classifiers in its local model using its data, but distills each other's 's knowledge at the same time. Our evaluation shows that deep classifiers can learn from shallow classifiers trained with otherwise unseen data, and that both classifiers actually help each other to improve the overall performance of the global model. We also analyzed the fundamental reason for the effectiveness of knowledge distillation in DepthFL, using an evaluation in general teacher-student environment.
Recently, InclusiveFL (Liu et al. 2022) proposed a kind of depth-scaled method with a similar intuition to ours, yet with less elaboration. We show that depth scaling alone in InclusiveFL without the companion objective of sub-classifiers cannot effectively solve parameter mismatches, and that its performance of deep classifiers is much lower due to self distillation. We employed the latest federated learning algorithm FedDyn (Acar et al., (02T) as the optimizer for proposed method. Although DepthFL is framework that focuses on resource heterogeneity of clients, we try verify if the FedDyn optimizer that focuses on data heterogeneity is applicable to DepthFF.seamlessly Our experiment shows that DepthFL works well for both resource heterogeneity and data heterogeneity.
Contributions To summarize, our contributions are threefold:
We present a new depth scaling method to create heterogeneous local models with sub-classifiers, which are directly supervised during training, thus no parameter mismatches when aggregated.
We show that knowledge distillation among the classifiers in local model can effectively train deep classifiers that can see only a limited amount of data.
We perform a comprehensive evaluation on the difference between depth scaling and width scal- ing using exclusive learning, and on the effectiveness of knowledge distillation in DepthFL.
2 RELATED WORK
2.1 FEDERATED LEARNING
Based on FedAvg McMahan et al. 2017). the standard federated learning method, there have been
many studies to solve various problems of federated learning (Li et al. 2020aa Kairouz et al.) 2019).
One main research field deals with efficient learning algorithms considering the non-llD distribution
of data among clients (Karimireddy et al. 2020 Li et al. [202 Wang et al. 2020 Acar et al.) 2021
Li al. 2020b). For example, FedDyn (Acar et al. (021) performs exact minimization by making the local objective align with the global objective, which enables fast and stable convergence. There are also studies to deal with heterogeneous, resource-constraiig
clients in federated learning. One popular method to create heterogeneous local models is prune the channels of global model
2

---

Published as a conference paper at ICLR 2023
Resdbock
Reydhack
Rostect
Mashowash
Nulhoooh
Variing
Lorda
Clonn
Diene 2
Olon
Clans
Cleett
Clont
de
Figure 1: Global model (ResNet) parameter W has kinds of local models as its subsets, distributed to m 6 heterogeneous clients. Each local model has multiple classifiers at different depths.
for width scaling (Diao 2021 Li al. 2021a Niu al. 2020 Horváth al. 2021
Hong
et al. 2022 Zhu et al. 2022). Based on the general layer-pruning methods (Chen & Zhao 2019
Sajjad al. 2023) that create shallow model. there is recent work (Liu et al., [202] prune
the layers of global model for tackling system heterogeneity, but it partially exploits the benefit
of depth scaling and knowledge distillation unlike DepthFL. (see Appendix A.3). Another direction
is utilizing knowledge distillation to reduce the burden on clients (He et al. (020) or to aggregate
heterogeneous local models (Zhang & Yuan 2021 Lin al. 2020 Separately, there are researches
to reduce the communication cost by compressing the local models (Rothchild et al. 2020 Had-
dadpour et al.) 2021; Reisizadeh et al.) 2020). There are also research directions on personalized
federated learning to handle heterogeneity of clients (Li et al. 2021c
Zhang al. 2021 Hanzely
cet al. 2020 T. Dinh et al. 2020).
2.2 KNOWLEDGE DISTILLATION
Knowledge distillation (Hinton 2015 was introduced as way of training a small student model with the help of a big teacher model. It was initially thought that the role of knowledge distillation is transferring high-quality, similarity information among categories, but researches are still on going to understand its impact such as the relationship with label smoothing regulariza- tion Yuaan et al. 2020 Tang al. 2020). As such, variants of knowledge distillation techniques have been proposed (Huag et al. 2021 Park et al., 2019) Zhang et al.) 2018: 2019). Deep mutual learning (Zhang et al. 2018) )) allows student models to mutually distill each other's knowledge with- out cumbersome teacher model. Self distillation (Zhang et al. 2019) distills knowledge within multi-exit network (Teerapittayanon et al.) 2016),, from its deep layers to shallow layers for higher performance, or for faster and accurate inference (Phuong & Lampert 2019). DepthFL also em- ploys self distillation within a local model, yet in an opposite direction, mainly for distilling from fully-trained shallow layers to partially-traine deep layers.
DEPTHFL METHOD
This section describes our proposed DepthFL. method in detail. We assume that a total of m hetero- geneous clients participate in federated learning, and that each client k has dy resource capability. Randomly selected P: E [m]]] clients participate in each communication round.
3.1 LOCAL MODELS
We create heterogeneous local models by pruning zero or more of the highest-level layers of the global model, instead of pruning some channels for all layers as in HeteroFL. Each client
3

---

Published as a conference paper at ICLR 2023
tribute to the training of the global model by selecting local model suitable for its resource amount, in a fixed or dynamic way. Since the highest-level layers are entirely prumed, an additional bottleneck layer is needed for local model to become independent classifier. This means that the global model should have a different classifier at a different depth. The structure of a global model that satisfies these conditions can be found in Deeply-Supervised Nets (DSN) (Lee et al.] (013). Each classifier shares the blocks that create feature maps, and an additional bottleneck layer is attached to each block so that the local models and the global model have several classifiers inside. Accordingly,
the local model parameters of the client whose resource capability is dy is Wid WgI: dg). and
has dy classifiers inside. The overall model structure can be found in Figure | DSN can provide integrated direct supervision to sub-classifiers in addition the output classifier. DSN exploits it for better performance, but DepthFL. exploits it for consistent training of sub-classifiers across all local models that includes it to fully exhibit its performance.
This way of creating heterogeneous local models is likely to make deep classifiers learn from partial amount of data, because only resource-rich clients can train local models with deep classifiers. On the other hand, shallow classifiers are likely be fully trained since many clients train local models with them. To mitigate the variance of the performance of different classifiers, it is natural to use their ensemble for inference. One interesting question is if partially-traine deep classifiers can really contribute to learning when many clients cannot afford a local model with them. In experimental result in Table 4 deep classifiers perform similarly even worse than shallow classifiers unless they receive a help, in the form of self distillation described below.
3.2 SELF DISTILLATION
To mitigate the low performance of deep classifiers, DepthFL utilizes the concept of self distillation. Several classifiers inside local model are trained through the cross-entropy loss with labels, as well as the KL loss with the output of other classifiers, as depicted in Figure T Previously, self distillation made shallow classifiers imitate deep classifiers (Zhang et al. 2019 Phuong & Lampert, 2019). Whereas, DepthFL makes deep classifiers imitate shallow classifiers, or more precisely, learn collaboratively. The local objective function of the client is follows:
du
du
du
LA
Σ 1=1,j
(KI(P))le)
(1)
d:
where Lic. is cross-entropy loss of ith classifier, and p; are softmax outputs of ith classifier's s logits. Although FjORD (Horváth et al., 202T) utilized self distillation already, the motivation of self distillation in FjORD was to allow for the bigger capacity supermodel to teach the width-scaled submodel. In contrast, the purpose of self distillation in DepthFL is that depth- help training of bigger capacity supermodels. Also, unlike distillation between width scaled models, which should run forward pass through teacher supermodel and student submodel independently. there is no overhead of distillation between depth scaled models except for the bottleneck layers because they share feature maps.
submodels
We use FedDyn (Acar et al. 2021) instead of FedAvg (McMahan al., 2017) as the default op- timizer for the above local objective function 1 to make a fast convergence even when there is a data heterogeneity of clients. When applying dynamic regularization, we replaced the client's local objective by the modified heterogencous local objective of] [ Also, V L ((f)) and h values required for dynamic regularization are used and updated in consideration of the heterogeneity of the local models. Therefore, the penalized local objective function of the client is as follows.
(Ø) (o) (VLx 0) + a - 0º112
where Ø is the local model parameters, V ( is the gradient of the local objective function in the previous round, and ge is parts of the current global model's parameters corresponding to the local model parameters 0. Considering the case where the complexity dę of the client changes dynamically depending on its current loads, VI (ff. is stored in the same shape the entire global model parameters, and only subset corresponding to the current local model parameters is used for
actual training. So, the value of LL (0E) ) is updated as in the following equation.
Lu (0 (0 ): d&) <- T (0 de)
ca 0¹)
(3)
4

---

Published as a conference paper at ICLR 2023
Algorithm 1: DepthFl.
Initialization: : 0¹, h1" O, ( = =0
Server executes:
DiientUUpdate(A, 8¹):
for round = 0, ......T-J.. do do
0x+ t 0e
P. < Random Clients
(O,( <- (f [: [x.
g++1+- +0 ht+1 + he
for local epoch E = 1.2.... E do for each mini batch b do
for each client k E Pt, and in parallel do
0" #+ *+00: dy]
LA = Σ Li + di 1 Σ
de
de
Σ
(p,ll)
0kt1 + Client. Update(k: 8¹
I dx) +- h [: dR) -a m (0 1 ( de))
-
Li(0) Lx(0) LV +) 110-
0 du] ( dz] + 0,+1
0% + oct1 - L (0
end
end
for each resource capability d: do
end LL (0( d+l <- (I+ de) -a(0k 0*)
g++ [d.]
g
[d:]
1 / [d,]
1
end end
return
0
Table 1: (Number of parameters) / [# of MACs] of local models according to division method
Model
Method
W?
W?
W/
d(W,) W7
HeteroFL (Width)
99.0 K [4.11 MJ
391 K [15.3 M]
877 K [33.6 MJ
1.56 M [58.9 MJ
ConvNet
DepthFL (Depth)
31,4 K [3.46 M]
178 K [23.6 M]
750 K [43.7 M]
1.94 M [62.6 M]
HcteroFL (Width)
713 K [35.5 M]
2.82 M [140 MJ
6.32 M [314 MJ
11.2 [557
Resnet-18
DepthFL (Depth)
480 K [167 M]
1.35 M [310 M]
3.84 M [450 M]
12.3 [585 M]
HctcroFL (Width)
1.36 M [73.6 M]
5.38 M [292 M]
12.1 M [655 M]
21.4 M [1.16 G]
Resnet-34
DepthFL (Depth)
0.60 M [243 M)
2.11 M [538 M)
9.38 M 1980 M]
22.6 M [1.19 G]
When the local models of different depths are aggregated into single global model, the local model parameters of the same depths will be aggregated together. This naturally makes shallow layers close to the input aggregate a large number of local model parameters, while deep layers close to the output aggregate small number of local model parameters. Algorithm [ shows the full algorithm. Replacing FedDyn optimizer by FedAvg is straightforward.
4 EXPERIMENTS
4.1 SETTINGS
Datasets and models We used MNIST, CIFAR-100, and Tiny ImageNet datasets for the image classification task, and WikiText-2 dataset for the masked language modeling task. CNN composed of 4 convolution layers, Resnet18, and Resnet34 models were used for MNIST, CIFAR-100, and Tiny ImageNet, respectively. The transformer model was used for Wikitext-2. We create four local models with DepthFL. and four local models with HeteroFL by dividing the channels into four equal parts following its division method. Table depiccss the model size and number of MACs for the four local models of HeteroFL. and DepthFL.. For example, the smallest model has only the classifier 1/4 in DepthFI.. while having 1/4 channels HeteroFI.. For inference, DepthFL. uses the ensemble of all internal classifiers, while HeteroFL uses the global model with all channels.
Local model If scaling is done as mentioned above, each depth-scaled local model would have fewer parameters and more MACs than the corresponding width-scaled one, since shallow layers often have fewer parameters yet require more computations. So. for the Resnet global model, the number of MACs decreases linearly the depth decreases, but the number of model parameters decreases more rapidly, as shown in Table|[[the global model of DepthFL has slightly more param- eters and MAC's than that of HeteroFL. due to the additional bottleneck layers). This means that the
5

---

Published as a conference paper at ICLR 2023
average communication overhead of depth-scaled clients would be lower than that of width-scaled clients, but it would be opposite for the average computation overhead. Also, depth scaling is less fine-grained than width scaling since depth scaling is more dependent on the layer structure of the global model, and may not be effective if the size of the bottleneck layer is too large. Despite these limitations, depth-scaled local models do not seriously affect the performance of the global model when aggregated, unlike width-scaled local models, as will be explained shortly.
Default Settings Unless otherwise stated, the same number of clients are allocated to each of the four different local models (i.e., 25% of the clients are allocated to each of a, b, c, and d local models in Table II. Also. the data is distributed in an IID manner, FedDyn is used as the optimizer, and randomly sampled 10% of the clients among a total of 100 clients participate in each communication round. When the data is distributed in a non-IID manner, as in FedMA (Wang et al. 20200). a Dirichler distribution Pa ~ (Br (B= = 0.5) was used to allocate Pa. ratio of data samples belonging to class e to client k.
4.2 COMPARISON WITH EXCLUSIVE LEARNING
We first evaluate if it is beneficial for resource-constraineed clients to participate in learning with depth-scaled local models even if they cannot accommodate given global model. For this eval- uation, we compare the accuracy of DepthFL with the accuracy of its exclusive learning, which excludes those clients who do not have enough resources run a given global model from learning. In DepthFL, d is the global model but we still allow those 75% clients whose resource cannot ac- commodate d to participate in learning. In exclusive learning, however, if d the global model, we allow only those 25% clients who can run d to participate in learning. Similarly, if is the global model, we allow those 50% clients who can run e (including those who can run but should run since is the global model now) to participate in learning. We also evaluate DepthFL without self-distillation and FedDyn (use FedAvg instead), which we call DepthFL(FedAvvgg by comparing with its corresponding exclusive learning. Finally, evaluate HeteroFL by comparing with its cor- responding exclusive learning. For a fair comparison, exclusive learning is trained and tested in the same way as the corresponding scaled method. For example, when comparing DepthFL with its ex- clusive learning, the local objective function of both includes the self distillation and regularization terms. Also, both are tested with the ensemble inference of the internal classifiers. It should be noted that we cannot compare the accuracy of DepthFL and HeteroFL directly, since their local models have a different size, so comparing only with their corresponding exclusive learning is meaningful.
Table 2 shows the results comparing HeteroFL, DepthFL(FcdAveg and DepthFL to their corre- sponding exclusive learning. In case of HeteroFL., exclusive learning with as the global model that prunes half of the channels, and with only 75% of the clients participating in learning, shows a tangibly better accuracy than HeteroFL in CIFAR-100. This means that although clients with insufficient resources can participate in learning on the HeteroFL framework, heterogeneous local models appear to deteriorate the global model and produce a worse result. DepthFL shows better result than any exclusive learning for MNIST and 100, but not for Tiny Imagenet. On the other hand, DepthFL performs better than any of exclusive learning in all datasets. This result indicates that DepthFL. can better train the global model using heterogeneous local models.
One question is why HeteroFL. shows a tangibly lower performance than exclusive learning, and why this is not the case for DepthFL To answer these questions, we measured the perfor- mance of each global sub-model of HeteroFL separately, by running 1/4, 2/4, 3/4, and 4/4 channels of the global model. We also measured the same for InclusiveFL (Liu et al., (002)) without momen- tum distillation and DepthFL the global model. Table shows the result compared to exclusive learning result of Table 2 (named Classifier 1/4-4/4). We can see that a global sub- mob-model of HeteroFL mostly performs worse than the corresponding exclusive learning model, while the global sub-models of (epthFL.(FedAAvg) per- form better than exclusive learning. In HeteroFL, for example, only 25% clients train Classifier 1/4 directly as in exclusive learning, while the rest 75% clients train it indirectly during the training of their 2/4, 3/4, or 4/4 Classifiers, so we suspect that this causes a parameter mismatch for Classifier 1/4 when being averaged, degrading the performance of the global model. To confirm this suspi- cion, evaluated a different version of HeteroFL called SHeteroFL. proposed in Split-Mix (Hong ctt al. 2022). where a client trains all affordable sub-models. For example, when a client with Clas- sifier 2/4 trains, it not only trains Classifier 1/4 indirectly when training Classifier 2/4, but trains
i.e., the performance of classifiers at 1/4, 2/4, 3/4, and 4/4 of

---

Published as a conference paper at ICLR 2023
Table Accuracy of the global model compared exclusive learning. 100% (a) exclusive learning means, the global model and every local model are equal to W model, and 100% clients participate learning. Likewise, 25% (d) exclusive learning means, the global model and every local model are equal to d(W,)) W model, and only 25% clients participate in learning.
Exclusive Learning
Scaling Method
Dataset
Global Model
100% (a)
75%
50% (c)
25% (d)
MNIST
99.33
99.38
99.39
99.41
99.19
HeteroFL
CIFAR-100
56.68
63.83
65.07
61.93
52.00
Tiny ImageNet
27.22
38.99
34.68
29.96
22.43
MNIST
99.43
99.29
99,35
99.39
99.25
DepthFL (FedAvg)
CIFAR-100
72.34
66.53
69.63
68.78
60.11
Tiny ImageNet
48.02
49.69
52.55
46.05
34.40
MNIST
99.51
99.25
99.41
99,41
99.34
DepthFI.
CIFAR-100
76.34
69.26
73.90
71.75
62.12
Tiny ImageNet
60.32
52.48
59.75
55.55
42.64
Table 3: Accuracy of global sub-models compared to exclusive learning on CIFAR-100.
Method
Algorithm
Classifier 1/4
Classifier 2/4
Classifier
Classifier 4/4
Exclusive Learning
63.83
65.07
61.93
52.00
Width Scaling
HeteroFL
50.41
55.05
57.71
56.68
SHeteroFL
63.57
66.14
65.44
63.98
Exclusive Leamming
66.53
67.03
66.47
57.68
Depth Scaling
InclusiveFL
47.25
53.36
58.57
60.10
SepthH1.(FedAyye
66.66
68.30
68.17
68.22
Classifier 1/4 directly. Table shows that SHeteroFL. differs from HeteroFI. in that the accuracy of its global sub-models is similar or higher than that of exclusive learning, as in DepthFL. This can also explain the better performance of DepthFL(FeddAg) than its exclusive learning. That is, when client trains its 2/4 Classifier, it also trains 1/4 Classifier with the companion objective, plained in Section InclusiveFL has no companion objective, showing a worse accuracy in Table (see also Appendix A.3). Actually, DepthFL does not require redundant training of sub-models when training their super-model, unlike SHeteroFL. Moreover, additional training of depth-scaled sub-mou uub-models can further improve the global model performance, unlike additional training of width- scaled sub-models (see Appendix I..I. DepthFL also works better than Split-Mix for the same amount of MACs (see Appendix [A.2). All these results indicate that DepthFL is more effective than other methods for handling parameter mismatches.
4.3 ABLATION STUDY
Knowledge Distillation To analyze the impact of mutual self-distillation on the performance of the global model, turn and off self distillation and measure the accuracy of the global model. We experiment with both IID and non-lID data distribution, whose results are depicted in Table Regardless of data distribution, shows that self distillation plays key role in enhancing the accuracy of deep classifiers. That is, deep classifiers tend to perform worse than shallow classifiers when self distillation off, yet they perform similarly or better in most cases when self distillation is on. 4able|4shows that self distillation improves the accuracy of shallow classifiers as well, which encouraging because resource- resource-constrainee constrained clients should do inference using shallow classifiers.
Maximum & Dynamic Complexity We evaluate why self-distillation enhances the accuracy of deep classifiers. Our conjecture was that fully-supervised shallow classifiers can train under- supervised deep classifiers that were partially trained with limited amount of data. For this eval- uation, we constructed a Maximum experimental environment where all clients can train the full global model (i.e., the model in Table Dp. We performed mutual self-distillation on the Maximum environment and measure the accuracy as in Table which includes the previous IID result of Table 4 (marked Fixed) for comparison. We can actually that the impact of self relf-distillation deep

---

Published as a conference paper at ICLR 2023
Table 4: Accuracy of the global model with/without self distillation for both IID/Non-IID data
Distribution
Datasct
KD
Classificr 1/4
Classificr 2/4
Classificr 3/4
Classifier 4/4
Ensemble
69.38
69.53
69.00
68.91
74.55
CIFAR-100
71.68 (+2.30)
73.89 (+4.36)
73.72 (+4.72)
73.56 (+4.65)
76.34 (+1.79)
IID
51.92
52.76
41.17
40.91
52.50
Tiny ImageNet
55.66 (+3.74)
58.50 (+5.74)
54.83 (+13.66)
55.20 (+14.29)
60.32 (+7.82)
66.68
67.68
67.66
67.25
72.64
CIFAR-100
69.61 (+2.93)
71.46 (+3.78)
71.52 (+3.86)
71,36 (44.11)
74.25 (+1.61)
Non-IID
50.33
50.76
44.11
43.61
53.14
Tiny ImageNet
53.84 (+3.51)
56.13 (+5.37)
54.28 (+10.17)
53.97 (+10.36)
58.99 (+5.85)
Table 5: Ablation study of the self distillation according to the resource complexity de distribution of the clients. Fixed complexity means that client's complexity de does not change from the initial value. Dynamic complexity means a client's 's dg value changes randomly every round. Maximum is the situation when all clients have sufficient resources, so all dę values are maximum. It also shows that DepthFL with self-distillation works well for the transformer model with WikiText-2.
Dataset
Complexity
KD
Classifier 1/4
Classifier 2/4
Classifier 3/4
Classifier 4/4
Ensemble
X
69.38
69.53
69.00
68.91
74.55
Fixed
/
71.68 (42.30)
73.89 (44.36)
73.72 (+4.72)
73.56 (44.65)
76.34 (+1.79)
X
69.33
72.44
73.97
73.92
77.14
CIFAR100
Dynamic
/
72.87(+3.54)
75.21(+2.77)
75.42(+1.45)
75.51(+1.59)
77.4%(+0.35)
X
68.99
72.36
74.42
74.37
77.48
Maximum
(2.330)3
75.04(+2.68)
76.47(42.05)
76.54(+2.17)
78.02(+0.54)
51.92
52.76
41.17
40.91
52.50
Fixed
55.66 (+3.74)
58.50 (+5.74)
54.83 (+13.66)
55.20 (+14.29)
60.32 (+7.82)
52.20
56.76
54.79
54.96
60.49
Tiny ImageNet
Dynamic
56.10 (+3.90)
60.35 (43.59)
60.02 (+5.23)
60.09 (45.13)
62.71 (+2.22)
51.36
57.30
53.27
54.05
59.88
Maximum
/
55.83 (+4.47)
60.62 (43.32)
59.88 (+6.61)
60.40 (+6.35)
63.47 (+3.59)
X
13.21
13.42
13.48
14.31
13.08
Fixed
/
12.99 (-0.22)
13. 24 (-0,18)
13.38 (-0.10)
13.99 (-0.32)
13.06 (-0.02)
WikiText-2 (Perplexity J)
X
13.21
13.31
13.38
13.80
13.02
Dynamic
/
12.97 (-0.24)
13.21 (-0,10)
13.29 (-0.09)
13.71 (-0.09)
13.03 (+0.01)
X
13.28
13.22
12.97
12.87
12.97
Maximum
/ 13.16 (-0.12)
13.16 (-0.06)
12.91 (-0.06)
12.86 (-0.01)
12.94 (-0.03)
classifiers in Maximum is tangibly smaller than in Fixed. This means that when resources are het- erogeneous, self-distillation achieves additional performance by transmitting the domain knowledge of shallow classifiers to deep classifiers. We also constructed a Dynamic experimental environment where the resources of the clients can change randomly in every round, yet the average amount of re- sources of those clients participating in each round is the same as in Fixed. So, unlike Fixed, deeper layers in Dynamic can learn from all data from every client as in Maximum, albeit less frequently. In 3 we can see that the impact of self-distillation on deep classifiers in Dynamic is similar to Maximum. This indicates that the lower accuracy of deep classifiers in Fixed is indeed due those unseen data of resource-constraing clients, and that self-distillation can alleviate the problem.
4.4 UNDERSTANDING SELF-DISTILLATION EFFECT OF DEPTIFL
This section attempts to understand the fundamental reason behind the effectiveness of self- distillation for depthFL. For this we evaluate knowledge distillation in general setup of teacher- student models in a centralized learning environment, except that the student model is trained with insufficient data (as the partially-traine, deep layers of DepthFL). We first evaluate if the existing analysis methods of knowledge distillation in (Tang et al. 2020). which are label smoothing (LS), gradient rescaling (KD-pt), and domain knowledge of class relationships (KD-sim), the effectiveness of self-distillation in DepthFL. (see A.6 for detailed explanation). The Resnet18 model is used as the student, while the Resnet101 model, which is fully trained with all data, is used as the teacher. Additionally, we analyze the effect of knowledge distillation by the poor teacher (PKD), which is trained only with 25% of the data, well as by the light teacher (LKD), which
explain
8

---

Published as a conference paper at ICLR 2023
Table 6: Top-1 accuracy of student on CIFAR-100 with a few partial knowledge distillation methods.
Data Amount Seen by Student
KD (Ace: 79.99)
KD-pt Tang
KD-sim Tang al. 2020
KD-pttsim Tang et. 2020
PKD (Ace: 58.14)
LKD
Basc
LS
2020
2020
(Ace: 72.19)
100%
77.60
79.47
78.52
78.85
78.50
78.80
72.12
77.18
50%
70.40
74.22
71.36
71.59
71.18
18
67.98
75.10
25%
59.10
66.26
60.31
61.11
61,90
62.05
63,30
71.46
Depth
(b) DepthFL without Distillation
800
80
75
75
70-
Test Acturacy
70-
80
60
0.00
0.00
0.00
1:00
0:00
0.00
0.00
1.00
0.10
0.10
0..0
0.06
10:40
0.10
0.10
0.40
0.40
35
0.25
0.25
0.40
0.40
0.10 1.330
0.40
0.40
0.30
0.10
50
Ensemblle
Eneemble
Depth of Class
Depth Classilier
Figure 2: Top-1 accuracy of four classifiers for four resource distribution ratios on CIFAR-100.
trained with all data but is sized smaller than the student model (as the fully-trained, shallow layers of DepthFL). As the light teacher of LKD, we use x0.25 depth-scaled local model a W of R8snet18. We perform these experiments with a variable amount of data the student learns as in Table 6 When the student model learns all data (100%), most effects of knowledge distillation could be explained by the three existing effects. However, when the student model learns partly (50% or 25%). the effect is not fully explainable by the composition of those three effects, neither by PKD. On the other hand, LKD is quite effective, achieving high accuracy even for 50% and 25% This means that even if the size of the light teacher is smaller than that of the student, the class relationship for each input instance can be transferred by the light teacher, which effectively helps for the generalization of the data iata-innuuficient student model. This additional role appears be the reason why self- eeff-disiillation in DepthFL. is effective, especially for data- inttfinnuuficient deep layers.
4.5 ROBUSTNESS TEST
The performance of DepthFL would inevitably be different depending on the distribution of cach client's resource capability. To evaluate the robustness of DepthFl., we changed the distribution of resource capabilities and measured the performance of the classifiers, as Figure [ As expected, as the ratio of cesstrce-constraineed clients increases, the performance of deep classifiers gets lower, even seriously when self distillation is off. With self distillation on, however, even if the deepest classifier is trained only by 10% of the clients, the performance drop is small, due to the help of other classifiers. Since shallow classifiers will be used for fast inference while an ensemble model will be used for high performance. deep classifiers do not always have be better than shallow ones. Appendix A.5.1 explains and evaluates whether deep classifiers are really needed for DepthFI..
5 CONCLUSION
We presented DepthFI., a new federated learning framework considering resource heterogeneity of clients. DepthFL. creates local models of different sizes by scaling the depth of the global model, and allocates them to the clients depending trains several classifiers within its local model, and at the same time, distills their knowledge with each other. As result, both deep classifiers trained with limited data and shallow classifiers trained by most clients can help one another to build the global model, with no parameter mismatch. We also evaluated depth scaling compared to width scaling thoroughly, and deff-distillttion in DepthFL.
their available resources. During local training, client

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IIPP) grant funded by the Korea government (MSIT) (No. 2021-0-00180, 25%) and (No. 2021-0-00136, 25%). by the ITRC (Information Technology Research Center) support program (LITP-021--0-018) 25%) supervised by the ITT,, and by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00208245, 25%).
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Con- ference on Learning Representations, 2021. URL. htttps://openrevieks net forum?
id=B7v4gMR679H
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMa- han, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. To- wards federated learning at scale: System design. In A. Talwalkar, V. Smith, and M. Zaharia (eds.), Proceedings of Machine Learning and Systems, volume 1, pp- mwl.g..msssss...mm.mm.mm.mmm.... httpll/ss.rrroppere//11//////
374-388, 2019. URL bd686fd6 wew.eeeeeeeee0000e0eeeelllll Paper. pdf
Shi Chen and Qi Zhao. Shallowing deep networks: Layer-wise pruning based on feature represen- tations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(12):30483305, 2019. doi: N0.11099TPAM..201.22555
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. In International Conference on Learning Represen- fations, 2021. URL htt.ttttts///peeeeeeteet
net/ Fo
Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. Fed- erated learning with compression: Unified analysis and sharp guarantees. In Arindam Baner- jee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artifi- cial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp- 2350-2358. PMLR, 13-15 Apr 2021. URL https: htttss:://pooeeeingss..ot wtwp:iis.s............ pta.s./rees//113// haddadpour la. .hmll
Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtarik. Lower bounds and optimal algorithms for personalized federated learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp- Curran Associates, Inc., 2020. URL
ngg .eeurnn wwwpp//ccccc.cccoo..o Paper.pdf.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Feder- ated learning of large cnns at the edge. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 14068-14080. Curran Associates, Inc., 2020. URL
neurips.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint 0503.00500.025:: 2015.
Junyuan Hong, Haotao Wang, Zhangyang Wang, and Jiayu Zhou Efficient split-mix federated learning for on-demand and in-situ customization. International Conference on Learning Rep- resentations, 2022. URL
net seret//orummicc. OLmakITKa
Samuel Horváth, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Donald Lane. FjORD: Fair and accurate federated learning under heterogeneous targets with ordered dropout. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL http:::///peeeevie.. net. fet./formmiin d=
10

---

Published as a conference paper at ICLR 2023
Zhen Huang, Xu Shen, Jun Xing, Tongliang Liu, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-Shann Sheng Hua. Revisiting knowledge distillation: An inheritance and exploration framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 3579-3588, June 2021.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Ad- vances and open problems in federated learning. arXiv preprint arXiv:1912.04977, 2019.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learn- ing. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Confer- ence on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5132-5143. PMLR, 13-18 Jul 2020. URL https:://ppoceeeee ngs. ml ptw.s/v119/ karimi reddy2aeeddyaaa... html
Chen- Yu Lee. Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- Supervised Nets. In Guy Lebanon and S. V. N. Vishwanathan (eds.), Proceedings of the Eigh- teenth International Conference on Artificial Intelligence and Statisties, volume 38 of Proceed- ings of Machine Learning Research, pp. 562-570. San Diego, California, USA, 09-12 May 2015. PMLR.
ml rr. press/ v38 v38/1ee//veeee ////eel55. 1tt.e111.mm...
Ang Li, Jingwei Sun, Pengcheng Li, Yu Pu, Hai Li, and Yiran Chen. Hermes: An efficient fed- erated learning framework for heterogeneous mobile clients. In Proceedings of the 27th Anmual International Conference Mobile Computing and Networking, MobiCom '21, pp. 420-437, New York, NY, USA, a. Association for Computing Machinery. ISBN 9781450383424. doi: 10.1 10.14453444999334432775 URLpttps: /toi. org/10. 11 3971999933442322 3483278
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10713- 10722, June 2021b.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):00-60, 2020a. doi: 10.1 N0.1009/MSP222022295774999
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Dhillon, D. Papailiopou- los, and V. Sze (eds.), Proceedings of Machine Learning and Systems. volume 2, pp. 429 450, 2020b. URL /ttto:://proceedd.. .ww.s...l.....m..mmmmmm.mmmm.... or. http:paper/222//f1ee 38a E 86 34b65d0 w1t.:333555000111113331e113e1110
pdf
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6357-6368 PMLR, 18-24 Jul 2021c. WuRLhtttss///proeeeeet
mlr. www.ppessswlll3
v / html
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. ran Associates, Inc., 2020. URL https: http://////.p..ceg../....o newrips cc/ cctpppppr///e///////////
Cur-
Fi filer//8afttsscccdcl222222966663
Ruixuan Liu, Fangzhao Wu, Chuhan Wu, Yanlin Wang. Lingjuan Lyu, Hong Chen, and Xing Xie. No one left behind: Inclusive federated learning over heterogeneous devices. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, pp. 3398-3406, New York, NY, USA, 2022. Association for Computing Machinery. ISBN doi: 10.1145/3554678.35522
9781450393850.
URL fttts://dci.orr///
3534678 3539086
11

---

Published as a conference paper at ICLR 2023
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-Effiiiii Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jemyy Zhu (eds.), Proceedings of the 20th International Conference on Artifi- cial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pp- 1273-1282. PMLR, 20-22 Apr 2017. URL https: https:///ccceedddeooooo ml w.w.p.eessvv... aocmahanla..hta
Chaoyue Niu, Fan Wu, Shaojie Tang, Lifeng Hua, Rongfei Jia, Chengfei Lv. Zhihua Wu, and Gui- hai Chen. Billion-scale federated learning on mobile clients: A submodel design with tun- able privacy. In Proceedings of the 26th Annual International Conference on Mobile Com- puting and Networking, MobiCom '20 New York, NY, USA, 2020. Association for Com- puting Machinery. ISBN 9781450370851. doi: 10.1145/3372224444111555 URL https:
3419188
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Pro- ceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Mary Phuong and Christoph Lampert. Distillation-based training for multi-exit architectures. In 2019 IEEE/CVE International Conference on Computer Vision (ICCV), pp. 1355-1364, 2019. doi: 10. 101110/9/CC222190000111
Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communication--ffiiien federated learning method with periodic averaging and quan- tization. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 2021-2031. PMLR, 26-28 Aug 2020. URL https: html
7/
www.ddiiin.smm.....mmmmmmmmm. mlr pppceedingg......rres/SSSSS
Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. FetchSGD: communication-effiice. federated learning with sketching. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 Proceedings of Machine Learning Research, pp. 8253 8265. PMLR, 13-18 18 Jul 2020. URL https: http:::://ppcceediiggge mlr www.mmesee..........1
rothchild20a html
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslay Nakov. On the effect of dropping layers of pre-trained transformer models. Computer Speech Language, 77:101429, 2023.
ISSN 0885-2308.
doi: http:///ooiorg/1000016.....c. sciencedirect. httpa//eeeeeee.cocccceeeee/eaeee http:///ccc......aaaa/a////////0
101429. URL https: htmpp::/wwrr /www.
Canh T. Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. In H. Larochelle, M. Ranzato. R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 21394-21405. Curran As- sociates, Inc., 2020. URL https: httphhts::://tttt httpps::/ppoceeedngss.. nwurpps......................... cttessccc/paaee wttpappee/22///1leeeeeeee htt.://8888aaaaeeee..0.11116666.
pdff
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed Chi, and Sagar Jain. Under- standing and improving knowledge distillation. arXiv preprint arXiv:2002.03532, 2020.
Surat Teerapittayanon, Bradley McDanel, and H.T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2464-2469, 2016. doi: 10.1 1..1109/CPPP.O WWW.CCCR..1....... 20166.99000.6.
Hongyi Wang, Mikhail Yurochkin, Yuckai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In International Conference Learning Represen- tations, 2020. httt:tttts::/pppmtee
net/ forum htt.:/orrrmmdmdsl1111lsssss
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
12

---

Published as a conference paper at ICLR 2023
Lan Zhang and Xiaoyong Yuan. Fedzkt: Zero-shot knowledge transfer towards heterogeneous device models in federated learning. arXiv preprint arXiv:2109.03375, 2021.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized fed- erated learning with first order model optimization. In International Conference Learning Representations, 2021. URL httppss//ppeerreeette
net wtwp://fonnnnnnnngo@0o00
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
Zhuangdi Zhu, Junyuan Hong, Steve Drew, and Jiayu Zhou. Resilient and communication effi- cient learning for heterogeneous federated systems. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th Inter- national Conference on Machine Learning, volume 162 of Proceedings of Machine Learning mmr.
Research, pp. 27504-27526. PMLR, 17-23 Jul 2022. URL 2ress/v162/zhu220 .tml
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 LIMITATION OF SHETEROFL
As mentioned in Section although SHeteroFL includes additional computation overhead of lo- cally training all possible sub-models, the performance of its global model that leverages all these sub-models can improve. However, we want to check whether additional training of sub-models of different widths is always helpful for the global model. We compared the performance of FedAvg, SHeteroFL, and (FedAv))) in the Maximum case introduced in Section 4.3 where all clients have enough resources to train the global model. Table shows the experimental result CIFAR- 100 using Resnet18 as the global model. SHeteroFL does not show better performance than FedAvg despite its additional training of width-scaled sub-models. On the other hand, DepthFL could im- prove the performance of the global model by additionally training depth-scaled sub-models, which in line with the result in Lee al. (2013). In other words, although SHeteroFL. could alleviate the problem of HeteroFL, DepthFL is a better approach to heterogeneous federated learning since it can better enhance the performance of global model with less computation cost,
Table 7: Accuracy of global sub-models in Maximum case.
Method
Classifier 1/4
Classifier 2/4
Classifier 3/4
Classifier 4/4
FedAvg
N/A
N/A
N/A
70.59
SHeteroFL
62.95
68.4
69.66
70.16
DepthFL (FedAvg)
66.65
70.11
73.15
73.36
A.2 COMPARISON WITH SPLIT-MIX (HONG ET AL. 2022)
Split-Mix provides customizable global model by mixing the base split models of the same sizes. While HeteroFL and DepthFL prune single global model to create local models of different sizes, Split-Mix can create a local model by combining small base models. To compare Split-Mix with DepthFL. we combine its base models to create four local models whose number of MACs is the same as that of the four local models of DepthFL.. For the CIFAR- dataset, Resnet18 was used as global model. The number of MACs and the number of parameters of four local models are shown in Table X As the base model Split-Mix, half of the channels of the global model were pruned. For fair comparison, DepthFL. was trained without regularization and self distillation.
Table 8: (Number of parameters) / [# of MACs] of local models according to division method
Model
Method
W!
W?
W?3
d(W,) W/
DepthFL (Depth) 480 K [167 MJ
1.35 M [310 MJ
3.84 M [450 MJ
12.3 M [585 M]
Resnet-18 18
Split-Mix (Width) 2.82 M [140 MJ
5.64 M (280 MJ
8.46 M (420 MJ
11.28 M [560 M]
The accuracy of DethpFL and Split-Mix in Fixed and Dynamic cases are shown Figure In Fixed, the learning curves of DepthFL and Split-Mix almost same. Since the number of parameters of local models of DepthFL is much smaller than Split-Mix, DepthFL is more efficient in terms of communication overhead. In Dynamic, DepthFL. shows faster convergence and higher accuracy than Split-Mix. The reason is that Split-Mix allows the client to train multiple base models alternately, so there is big difference between Fixed and Dynamic, whereas in DepthFL, the deep classifier can train more data in Dynamic, as explained in Section 4.3
A.3 COMPARISON WITH INCLUSIVEFL LIU ET AL. 2022
Recently, InclusiveFL. (Liu et al. 2022) proposed a kind of depth-scaled method for heterogeneous federated learning, However, there are two major differences from DepthFL. First, unlike DepthFL, InclusiveFL trains only the last classifier of the local model without training the sub-models, As we saw in Table 3 training sub-mu models during local training has a decisive impact on the performance of global sub-models, hence the overall performance (its ensemble result is 63.79, lower than the
14

---

Published as a conference paper at ICLR 2023
Fixed
Dynamic
0.
0.5
0.5
0.4
0.4
0.3
02
02
Split
DepchFLiFeckwa)
DepthFLiFeckvr)
0.1
0.1
203
400
800
802
1000
200
400
000
800
1000
Round
Round
Figure 3: Learning curves for DepthFL and Split-Mix
ensemble result 72.34 of DepthFl.). That is, similar to the comparison result between SHeteroFL. and HeteroFL. DepthFL has superior performance of sub-classifiers compared to InclusiveFL, and this has a direct impact on the performance of the global model. Actually, unlike DepthFL, InclusiveFL uses only the output of the last classifier during inference, so the performance of the sub classifiers is not that important, and InclusiveFL. has no overhead for bottleneck layers. Nevertheless, the performance of Classifier 4/4 InclusiveFL is even lower than exclusive learning 1/4-3/4, which indicates that training all sub-models in the local model using the companion objective is critical.
Second, since InclusiveFL. locally trains only the last classifier in the local model, knowledge dis- tillation between sub-models is impossible. InclusiveFL uses a special technique called momentum distillation to transfer the knowledge of a larger model to a smaller model, which has a limitation since the size of each layer must be kept constant. Also, its role is different from self-distillation in DepthFL., which transfers knowledge of a smaller model to a larger model. In summary, since InclusiveFL does not train sub-models in the local model through companion objective nor employ self distillation, InclusiveLL does not fully exploit the advantages of depth scaling unlike DepthFL..
A.4 OPTIMIZER
DepthFL employs FedDyn (Acar et al., 2021) as the default optimizer. To analyze the impact of the optimizer, we replaced FedDyn with FedAvg (McMahan et al., 2017). And, we performed the same experiments in Table g and T whose results are shown in Table Fanleaanndd and Table I0 respectively. Even if FedAvg is used as the optimizer, the overall trend of the experimental results is almost the same, but we can see that the overall accuracy is lowered. Also, we can see that the impact of self-distillation is reduced compared to when FedDyn is used.
Table 9: Accuracy of FedAvg global model with/without self distillation for both IID/Non-IID data
Distribution
Dataset
KD
Classifier 1/4
Classifier 2/4
Classifier 3/4
Classificr 4/4
Ensemble
66.66
68.30
68.17
68.22
72.34
CIFAR-100
68.55 (+1.89)
70.23 (+1.93)
71.21 (+3.04)
71.10 (+2.88)
72.96 (+0.62)
IID
49.38
48.76
38.18
38.44
48.02
Tiny ImageNet
/
51.29 (+1.91)
52.56 (+3.80)
48. 99 (+10.81)
49.66 (+11.22)
53.76 (+5.74)
X
65.82
66.22
66.22
65.65
70.63
CIFAR-100
/
67. (+1.32)
68 (+2.19)
69.38 (+3.16)
68.84 (+3.19)
71.48 (+0.85)
Non-IID
X
47.01
46.55
39.36
38.74
47.67
Tiny ImageNet
49.05 (+2.04)
49.73 (+3.18)
47.03 (+7.67)
46.86 (+8.12)
51.67 (+4.00)
15

---

Published as a conference paper at ICLR 2023
Table 10: Ablation study of self-distillation according to resource complexity de distribution.
Dataset
Complexity
KD
Classifier 1/4
Classifier 2/4
Classifier 3/4
Classifier 4/4
Ensemble
X
66.66
68.30
68.17
68.22
72.34
Fixed
/
68.55 (+1.89)
70.23 (+1.93)
71.21 (+3.04)
71.10 (+2.88)
72.96 (+0.62)
X
67.09
70.94
73.32
73.48
75.46
CIFARION 100
Dynamic
/
69.18(42.09)
72.51(+1.57)
(4.28))0.96)
74.34(40.86)
75.04(-0.42)
X
66.65
70.11
73.15
73.36
75.05
Maximum
/
(9.07)(2.442)
71.92(+1.81)
(4.77(+1.622)
75.09(+1.73)
75.41(+0.36)
X
49.38
48.76
38.18
38.44
48.02
Fixed
/
51.29 (+1.91)
52.56 (+3.80)
48, (+10.81)
49.66
(+11.22)
53.76 (+5.74)
49.41
53.29
48.76
49.29
54.81
Tiny ImageNet
Dynamic
/
52.06 (+2.65)
55.25 (+1.96)
54.79 (+6.03)
55.08 (+5.79)
57.40 (+2.59)
X
48.58
52.82
46.97
48.06
53.77
Maximum
/
51.45 (+2.87)
55.09 (+2.27)
52.98 (+6.01)
53.97 (+5.91)
56.84 (+3.07)
X
14.28
14.47
14.92
15.64
14.19
Fixed
/
14.26 (-0.02)
14.44 (-0.03)
14.85 (-0.07)
15.47 (-0.17)
14.21 (+0.02)
WikiText-2 (Perplexity D)
X
14.27
14,40
14,85
15.26
14.10
Dynamic
/
14.23 (-0.04)
14.39 (-0.01)
14.80 (=0.05)
15.19 (-0.07)
14..4
X
14.25
14,32
14.67
14.91
13.96
Maximum
/ 14.23 (-0.02)
14.37 (+0.05)
14.69 (+0.02)
14.92 (+0.01)
14.06 (+0.10)
A.5
MORE ABLATION STUDIES
A.5.1 DEEP CLASSIFIERS ARE REALLY NEEDED?
Table showed that the performance of the deepest classifiers is similar or slightly lower than that of shallow classifiers even with self-distillation. So, one might question if deep classifiers are really needed for resource-heteree heteroeeeeeero federated learning, since they are likely have fewer clients to train although they allow more generalization of the model. That is, if most clients have insufficient resources, deeper layers might not help; otherwise, they might contribute to enhancing the performance of the global model. On the other hand, as in the Dynamic case in Table when the client resources change dynamically over time so that all clients can train deep layers even occasionally, deep layers can have a higher performance.
In reality, the performance of classifiers with different depths would inevitably be different depend- ing on various factors such as the distribution of client resources, the nature of the learning task, the number of data each client has, and the size of global model/sub-modee deep classifiers are really necessary by performing an experiment excluding deep classifiers, whose result is in Figure For example, DepthFL (exclude #4 Classifier) reduces the depth of the global model 3/4, that those 25% clients whose resource can train Classifier 4/4 trains only up Clas- sifier 3/4. We conducted the experiment with big and small global models (Resnet1 8 and ConvNet) for two learning environments (Fixed and Dynamic) on CIFAR-100. We can see that if the size of the global model is smaller compared to the task, and if the client resources change dynamically so the deep classifier can see more data, deep classifiers gets more important for high performance. Actually, this paper presented general framework that can flexibly deal with such diverse situations.
We tried to test whether
A.5.2 NON-IID DEGREE AND NUMBER OF CLIENTS
We performed ablation study for diverse non-lID IID degrees and for different number of clients, whose results are Table I and Table 12 respectively. They show a similar behavior as previously.
A.6 PARTIAL KNOWLEDGE DISTILLATION METHODS
The experiment in Section was conducted as follows. In KD-pt method, we synthesize teacher distribution pe as n.dh Pt if I, (1 pt) - 1) otherwise, where Pt is prediction on ground truth class from the teacher's probability distribution. In KD-sim method, we synthesize teacher distribution o sin as the softmax over cosine similarity between the teacher's last logit layer's weights: paim coftmax(relu((( (VT))/B), where W E kxd is the normalized logit layer
16

---

Published as a conference paper at ICLR 2023
Resnet18 CIFAR100 (Fixed)
Conviet CIFAR100 (Fied)
0.70
0.60
0.55
0.33
0.50
DepthFi
DeptiFi
DepthFL (exclude e Classifieri
Depthe ferddude clasaffier)
0.45
Depthl' (exclude
045
Depthll
DepthiL (exclude * 2,3 classifier)
DepthiL ferclude 2,3.4 Classifier)
400
930
1000
200
800
Round
Round
Resmet18 CIFAR10D (Dynemic)
ConvNet CIFAR1DO (Dyne mic)
0.80
0.75
0.65
0.55
DepthFL
DepthFL
0.30
DepthFl (exclude # Clessifier)
DepthfL fexcuude 4 Classifier)
(exclude (exclue # 3,4 Clmsifirr
DeptifLL (exclude 3,4 Cassilier)
DeptheL (exclude classifier)
Depthe fexdude 2.3.4 Classifier)
0.40
040
Bound
Round
Figure 4: Learning curves for DepthFL without deep classifiers
Table 11: Ablation Study
Data distribution
Dataset
Distribution
KD
Classifier 1/4
Classifier 2/4
Classifier 3/4
Classifier 4/4
Ensemble
69.38
69.53
68.91
74.55
IID
71.68 (42,30)
73.89 (+4.36)
73.72 (+4.72)
73.56 (+4.65)
76.34 (+1.79)
66.68
67.68
67.66
67.25
72.64
Din(0.5)
/
69.61 (+2.93)
71.46 (+3.78)
71.52 (+3.86)
71.36 (+4.11)
74.25 (+1.61)
CIFAR-100
65.69
66.83
66.68
66.24
71.54
Din(0.3)
68.46 (42.77)
70.82 (43.99)
70.92 (+4.24)
70.57 (44.33)
73.51 (+1.97)
60.97
62.35
61.97
60.79
67.35
Dir(0.1)
63.88 (+2.91)
65.06 (+2.71)
64.93 (+2.96)
64.14 (+3.35)
68.24 (+0.89)
Table 12: Ablation Study # Clients
Dataset
Clients
KD
Classifier 1/4
Classifier
Classifier 3/4
Classifier 4/4
Ensemble
x
69.20
68.95
68.53
68.59
74.39
50
/
71.8% (+2.68)
72.74 (+3.79)
72.96 (+4.43)
73.02 (+4.43)
76.12 (+1.73)
X
69.38
69.53
69.00
68.91
74.55
CIFAR-100
100
/
71.68
(+2.30)
73.89 (+4.36)
73.72 (+4.72)
73.56 (+4.65)
76.34 (+1.79)
X
68.59
69.18
68.57
68.88
73.89
200
/
71.82 (+3.23)
72.58 (+3.40)
73.37 (+4.80)
73.23 (+4.35)
75.47 (+1.58)
weights, w is the f-th row of W corresponding to the ground truth, and a, 3 are hyper-parameters for resolution of cosine similarities.
17

---

Published as a conference paper at ICLR 2023
A.7 LEARNING CURVES
The learning curves of Table for comparing HeteroFL, DepthFL(FedAvv), and DepthFL with their corresponding exclusive learning study of knowledge distillation are in Figure 6
Figure 5
The learning curves of Table for the ablation
The x-axis is the communication round, and the
y-axis is the moving average the accuracy.
(e) CIFAR1CO Heterof us Exclusive Learning
(b) liny ImageNet
HeterohL vs Exclusive Learning
0.40
0.35
0.
0.30
0.25
NeterofL
a-100%
0.20
0.3
b-75%
6.75%
50%
015
50%
d-25%
d-25%
200
400
000
800
200
400
600
800
Round
Found
(c) CIFAR100 DepthFL vs Exclusive Learning
(d) Tiny Imayllaaeeett--- DepthFLi(eedapy vs Exclusive Learning
07
0.6
0.40
0.5
0.35
DopthFL
DeptiFf
a-100%
5 100%
0.3
65%
0.25
6.75%
c 50%
d-25%
0.20
200
400
600
800
200
400
600
600
Hound
Round
(e) CIFAR100 DepthFL Exclusive Learning
(f) Tiny-lmage DepthFL Exclusive Learning
0.7
0.5
0.6
DopthFL
DepthFL
100%
100%
0.3
5050%
02
50%
d-25%
d-25%
0.2
Round
Sound
Figure 5: Comparative experiments with exclusive learning
18

---

Published as a conference paper at ICLR 2023
(al CIFARLOO(IID) without seif Distillation
(b) CIFARIOO(IID)
0.7
0.6
0.5
DepthPL
DepthfL
0.4
1/5 Llassifier
1/4 Llassifier
Classifier
2/4 Classifler
Classifier
Classifier
0.3
4/4 Classifier
4/4 Classifier
Round
Round
Tiny-ImageNet(il))
stillation
(d) Tiny-ImageNet((I))
0.55
0.50
0.40
0.4
0.30
DepthFL
DepthfL
1/4 Classifier
1/4 Clazsifier
2/4 Classifier
Classifier
0.20
Classifier
3/4 Classifier
4/4 Closzifler
4/4 Classifier
Rounci
Round
(e)
DII without
CIFAR1OC(NONIID)
0.6
0.5
0.4
DepthFL
DepthFL
0.3
Classifier
0.
1/4 Classifier
Classifier
2/4 Classifier
02
3/4 Classifier
3/4 Classifler
Classifier
4/4 Classifier
200
400
800
Round
Round
(g) (onnlll without Self Distillation
0.
0.5
0.4
0.3
DepthFL
DepthFL
Classilier
Classifier
2/4 Classifier
2/4 Classifler
3/4 Classifier
3/4 Classifier
0.1
Classifier
01
Classifier
0
200
400
800
800
200
400
800
800
Round
Round
Figure 6: Ablation study of self distillation
19

---

Published as a conference paper at ICLR 2023
B HYPERPARAMETERS
Most the hyperparameters used our experiments are depicted in Table 13 We set both a, B hyper-paramelers to 0.3 for KD-sim (Tang al. 2020) experiments. In the masked language mod- eling task with the transformer model, we randomly selected 15% of the input tokens, Among the selected tokens, 80% were modified [mask] tokens, 10% to random tokens, and 10% to remain unchanged.
Table 13: Hyperparameters and model architecture used in experiments
Data
MNIST
CIFARI00
LmyyImmgeNNe
WikiText-2
Model
ConvNet
Resnet-18 [64, 128, 256, 512]
Resnet-34 [64, 128, 256, 512]
Transformer
Hidden size
[64, 128, 256, 512]
[512, 512, 512, 512] 1
Local Epoch E
Local Batch Size B
64
64
50
Optimizer
SGD
Momentum
Weight decay
le-3
Temperature
a (FedDyn)
0.1
Consistency rampup
300
300
300
120
Communication rounds
1000
1000
1000
150
Learning rate
0.1
Learning rate decay
0.998
0.998
0.998
0.98
Embedding Size
512
Number of heads
8
Drupout
N/A
0.2
Sequence length
64
20


Reviews:
[{"review_id": "zSwfivm93G", "reviewer": "Reviewer_Gy8o", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose Depthwise Federated Learning (DepthFL) framework (includes mutual self-distillation)  to ensure that the global model accuracy improves compared to exclusive FL (excluding resource-constrained clients which can not train global models)) when performing aggregation of local models in FL on heterogeneous clients, especially, resource-constrained devices. \n- DepthFL is based on depth scaling in contrast to conventional approaches on width scaling, the latter suffers from degrading accuracy when compared to exclusive FL.\n- DepthFL constructs a global model that has several classifiers of different depths. It prunes the highest-level layers of the global model to create local models with different depths based on the client's available resources, thus with a different number of classifiers.\n- Each client not only trains the classifiers in its local model using local data but also distills the knowledge  across classifiers at the same time\n-  For inference, DepthFL uses the ensemble of all internal classifiers while HeteroFL uses the global model with all channels", "summary_of_the_review": "This paper tackles an important problem of FL on heterogeneous clients. The idea is simple to understand which is to have clients with different resource availabilities have local models of varying capacity ( here, a depth-wise pruned version of the global model). However, the paper is written in a way that shows it as a limited novelty by applying Deeply-Supervised Nets (Lee et al., 2015) and deep mutual learning (Zhang et al., 2018). Moreover, an important extensive comparison with InclusiveFL is missing from the main body (though a section exists in Appendix) which is closest in design to DepthFL and has the same baseline comparisons used in this work. \n\nHowever, I will be willing to update my score based on overall discussion and clarifications of the issues raised.\n\n---\nEdit: I have read the authors' responses and increased my score.", "strength_and_weaknesses": "**Strengths:**\n- The application of depth-wise pruning to tackle resource heterogeneity in FL\n- Distillation from shallow to large classifier models via deep mutual learning\n- The main ideas in the paper are easy to understand for the reader\n- Extensive experiments with ablation studies that help understand the intuition behind design choices\n\n\n**Weaknesses:**\n- Related study is missing discussion on layer-pruning methods\n- Novelty of the proposed method appears to be limited as the application of Deeply-Supervised Nets (Lee et al., 2015) and deep mutual learning (Zhang et al., 2018) for FL to improve upon the layer-pruning method (InclusivFL). It might be better to improve Section 1 to clarify the novelty.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hj978H1-9N5", "reviewer": "Reviewer_nCtb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes DepthFL, an approach that enables resource-constrained clients to participate in Federated Learning (FL) of a global model that is larger than these devices. DepthFL defines shallower versions of the global model for simultaneously training, whereby each client trains a local model of a depth appropriate for its resources.  DepthFL employs self-distillation of the highly trained shallow layers to address the insufficient training of deeper layers. The experimental results show improved results for both the global model and the shallow local models compared to excluding resource-constrained devices. ", "summary_of_the_review": "The paper convinced me that local models of varying depths and self-distillation is an effective way to improve performance and usefulness of FL on heterogeneous clients. ", "strength_and_weaknesses": "The paper addresses an important problem by enabling resource-constrained clients in heterogeneous FL. I found the two techniques of varying depths to match client resources and self-distillation of the deeper layers to be quite intuitive. I also found the evaluation section to be well organized, motivated, and presented. \n\nThe only weakness is that I would have liked to see an evaluation of a scenario where 100% of the clients could participate in training the global model, as that could help to understand the upper bound performance of these tasks. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "_gB9a_Qs7h", "reviewer": "Reviewer_qicp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a new Federated Learning (FL) approach for tackling resource heterogeneity. In their proposed scheme, local models are scaled up horizontally, and FL clients can choose to learn local models with different depths based on their needs. One novelty resides in their design of the reverse direction knowledge distillation, where knowledge is distilled from shallower layers, which were trained by more clients, to deeper layers. As a tradeoff, each layer is attached by a classifier head.", "summary_of_the_review": "A technically sound work towards addressing system heterogeneity in FL by layer-wise scaling.", "strength_and_weaknesses": "Pros:\n\n+ Techically sound approach towards tackling system heterogeneity in FL.\n+ Novelty on reverse direction knowledge distillation; authors also provided empirical analysis on its effects.\n\nCons:\n* Extra computation overhead exists in 1) training multiple classifier heads and 2) self-distillation among layers. These classifiers are also required to be transmitted during FL loops.\n* Resemblance to early-exit models: The design logic largely reminds me of the early-exit networks (e.g. [1]). Authors are encouraged to cite work along this line and discuss the difference between the proposed work and early exit models.\n* Non-comprehensive baselines: authors mentioned different FL approaches of width-based pruning in the related work section, but only FedHetero has been extensively evaluated as the only baseline. When analyzing the lower performance of FedHeteo compared with exclusive learning, I suggest authors study and discuss more related work that enables split learning in FL, such as [2] and [3] which tackled the issue that exists in FedHetero.\n \n\n\n[1] BranchyNet: Fast inference via early exiting from deep neural networks.  ICPR 2016.\n[2] Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. NIPS 2021.\n[3] Resilient and Communication Efficient Learning for Heterogeneous Federated Systems. ICML 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Fu9XHX3BCR-", "reviewer": "Reviewer_jAAa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to address the resource challenge of federated learning. Instead of training the same server model across all clients, it proposes scaling the server model along the depth dimension instead of the width dimension to meet the resource limitation of each client. It also found that distilling knowledge from the shallower part to the deeper part is essential to improving the accuracy of the final model ensemble.", "summary_of_the_review": "Per the discussion in the other sections, I think the novelty of this paper is somewhat limited, and the concerns mentioned in the weakness section make me skeptical about the correctness of the claim that depth-wise scaling is better than width-wise scaling. I will be happy to change my rating if the concerns can be resolved in the rebuttal.", "strength_and_weaknesses": "Strength:\n- This paper address an important issue of federated learning that many clients are constrained by the available resource and cannot train the whole server model.\n- This paper shows that depth-wise scaling is better than width-wise scaling for federated learning.\n- This paper shows that self-distillation from the shallower part to the deeper part can help improve the final accuracy.\n- Using the almost free model ensemble (under this paper's experimental setting) to enhance the accuracy is clever.\n- The experiments are extensive and informative, and I appreciate the authors' efforts in designing and running them.\n\n==============================================================================\n\nWeakness:\n- I noticed that in Table 2 the accuracy of exclusive learning of DepthFL (FedAvg) is consistently higher than that of HeteroFL. The sub-model of HeteroFL is thinner but deeper while the sub-model of DepthFL (FedAvg) is shallower but wider. Empirically, the former will perform better than the latter given the same model size. These counter-intuitive results make me wonder whether the gain comes from other sources instead of doing depth-wise scaling, such as more intermediate supervision points (like SHeteroFL) or unfair comparison (as stated in the last sentence of the first paragraph in Sec. 4.2).\n- Does HeteroFL use the same architecture as DepthFL with the intermediate supervision points? If not, DepthFL is actually larger and more complicated than HeteroFL.\n- Missing the comparison to Federated Dropout. HeteroFL sounds like a special case of Federated Dropout. HeteroFL drops the neurons in a fixed order while Federated Dropout drops the neurons randomly. I wonder whether the results will change if HeteroFL drops different 25% for each client and for each round instead of the same 25%.\n- The justification of why depth-wise scaling is better than width-wise scaling in the last paragraph of Sec. 4.2 sounds weird to me. The claim is that the worse accuracy of width-wise scaling is due to the fact that 25% clients train classifier 1/4 directly and 75% clients train classifier 3/4 indirectly. I think it is the same for the proposed depth-wise scaling. For example, the first 25% layers are also trained by 75% clients indirectly due to backprop. Why can this justify the accuracy difference?\n\nOther comments:\nBecause it uses a branchynet-style network to resolve the resource issue, it requires adding multiple intermediate supervision points with decoders or classifiers. If the decoders and the classifiers are heavy, adding them may nullify the saving in the reduced encoder and increase the complexity of the global model. Some example applications are image reconstruction or speech recognition.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster