Aggregating Capacity in FL through Successive Layer Training for Computationally-Constrained Devices
Authors: Kilian Pfeiffer
Ramin Khalili
Joerg Henkel
Venue: NeurIPS 2023 poster (2023)
Keywords: Federated Learning
Memory
Resource Constraints
TL;DR: layerwise training works better than training model subsets for memory-constrained devices in federated learning.

Abstract: Federated learning (FL) is usually performed on resource-constrained edge devices, e.g., with limited memory for the computation. If the required memory to train a model exceeds this limit, the device will be excluded from the training. This can lead to a lower accuracy as valuable data and computation resources are excluded from training, also causing bias and unfairness. The FL training process should be adjusted to such constraints. The state-of-the-art techniques propose training subsets of the FL model at constrained devices, reducing their resource requirements for training. However, these techniques largely limit the co-adaptation among parameters of the model and are highly inefficient, as we show: it is actually better to train a smaller (less accurate) model by the system where all the devices can train the model end-to-end than applying such techniques. We propose a new method that enables successive freezing and training of the parameters of the FL model at devices, reducing the training’s resource requirements at the devices while still allowing enough co-adaptation between parameters. We show through extensive experimental evaluation that our technique greatly improves the accuracy of the trained model (by 52.4 p.p. ) compared with the state of the art, efficiently aggregating the computation capacity available on distributed devices.

A Aggregating Capacity in FL through Successive Layer Training for Computationally-Costttaanee Devices
Kilian Pfeiffer
Ramin Khalili
Karlsruhe Institute of Technology
Huawei Research Center Munich
Karlsruhe, Germany
Munich, Germany
kilian. pfeiffer@kit.ed
ramin. kamin Milalllonn @huawei.com
Jörg Henkel
Karlsruhe Institute of Technology
Karlsruhe, Germany
henkel0kit. lekeelekiteedu
Abstract
Federated learning (FL. is usually performed on resource-constraiee edge devices, e.g., with limited memory for the computation. If the required memory train a model exceeds this limit, the device will be excluded from the training. This can lead to a lower accuracy as valuable data and computation resources are excluded from training, also causing bias and unfairness. The FL training process should be adjusted to such constraints. The state-of-the-ar techniques propose training subsets of the FL model at constrained devices, reducing their resource requirements for training. However, these techmiques largely limit the co-adaptation among parameters of the model and are highly inefficient, as we show: it is actually better train a smaller (less accurate) model by the system where all the devices can train the model end-to-end than applying such techniques. We propose a new method that enables successive freezing and training of the parameters of the FL model at devices, reducing the training's 's resource requirements at the devices while still allowing enough co-adaptation between parameters. We show through extensive experimental evaluation that our technique greatly improves the accuracy the trained model (by 52.4p.p.) compared with the state the art, efficiently aggregating the computation capacity available on distributed devices.
Introduction
Federated learning (FL) has achieved impressive results in many domains and is proposed for several use cases, such as healthcare, transportation, and robotics processed centrally but usually on resource-constraine edge devices, training machine learning (ML) models impose large computational burden on these devices 77. Additionally, FL requires communication, specifically exchanging ML model parameters from the devices to centralized entity for aggregation. Extensive research has been done to lower the communication overhead required for FL, e.g., on the use of quantization in the communication [8, 9 or sketched updates [10). Similarly, techniques such as partial updates [1 asynchronous aggregation [12 [ and tier-based aggregation [4 [5] have been proposed to lower and account for varying computational throughput. While constrained computation throughput and communication capabilities can slow down FL convergence, high memory requirements for training that are imposed on devices exclude devices completely from the FL system. This is, for example, the case in Google GBoard [16], where devices that do not have 2GB of memory for training removed. Excluding devices from training
5 6). As data in FL is not
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

lowers the reachable accuracy, as fewer devices participate in the training, also resulting in bias and unfairness [77]
Several techniques have been proposed to tackle these constraints, where the main idea is to train lower complexity submodel on the devices and embed the trained submodel into the full higher- capacity server model. A submodel is typically created by scaling the width of the neural network (NN), e.g., using subset of convolutional filters per NN layer. There exist several variations of the technique 9 18
19 20] In particular, Caldas et al. [9] propose Federated Dropout (FD), which randomly, per round and per device, selects NN filters that are trained. Alam et al. 22 propose FedRolex, a sliding window approach, where all devices train the same submodel, and in each FL round, the used filter indices are shifted by one. While both these techniques allow training within given memory constraints, our results show (Fig. D that they perform worse than a straightforward baseline, i.e., using a smaller NN model that can be trained by all devices end-to-end. We evaluate CIFAR1O, 10, FEMNIST, and TinylmageNet in an FL. setting using ResNet and scale the width of the NN down s.t. we achieve a 8x reduction in training memory. We observe that training the same small model at all devices outperforms FedRolex and FD w.r.t to the final accuracy and convergence speed (we expect similar results for other subset-derived techniques), especially when enforcing a 4x and 8x memory reduction (as also evaluated in (20]
CIFAR10/ResNe120
FEMNIST/ResNN1120
ThaylmageNe/ReeNeeu4
Small Model
Serreeyyy [%)
80
80
40
FedRolex
20
Recurree
60
40
60
20
FD g
20
20
40
0 8x
8x
2X
1x
8x
2X
1x
2x
1x
memory reduction
memory reduction
memory reduction
Figure 1: Accuracy of FedRolex 220 and FD [9] compared to small model using different NN topologies and datasets after /500/ 1000 FL rounds. In the case of 1x, both techniques are equivalent to vanilla Federated Averaging (FedAvg). Hyperparameters of the experiments are given in Section 3
Our results indicate that applying these techniques is rather harmful. This is as a large part of filters/parameters has to be dropped during each round of training at each device, extremely limiting the co-adaptation between parameters. Hence, the gradients for the subset of parameters that are trained on devices are calculated without considering the error of the parameters that reside on the server (more details in Appendix AA. Motivated by these observations, we propose new technique that enables successive freezing and training of the parameters of the FL model at devices, reducing the training's 's resource requirements the devices while allowing a higher co-adaptation between parameters. Instead of switching between subsets of the model on an FL-round basis, we train the same parameters for several rounds and successively switch to a larger model. To obey the same memory constraints as in [9 200, we train early layers using the full width while utilizing a scaled-down NN head. We then freeze the early layers and expand the head layers' width. By freezing early layers, no activation has to be kept in memory, hence, we lower the memory footprint. But still, the error of these frozen parameters included in the calculation of the gradient of the new subset of parameters. We apply this technique successively till all parameters of the model are trained.
In summary, we make the following novel contributions:
We empirically show that employing current sfate-of-the-ar techniques, FD 19 and Fe- systems can actually hurt the performance.
dRolex (20]. for memory-constraine
We propose a novel training scheme called Successive Layer Training (SLT which ad- dresses the shortcomings of previous techniques by successively adding more parameters to the training, successively freezing layers, and reusing a scaled-down NN head.
Our evaluation of common NN topologies, such as ResNet and DenseNet, shows that SLT reaches significantly higher accuracies in independent and identically distributed (iid) and non-iid CIFAR. FEMNIST, and TinyImageNe training compared to the state of the art. Also, SLT provides a much faster convergence, reducing the communication overhead to reach certain level of accuracy by over 10x compared with FD and FedRolex. The same
'The source code of SLT is available at htthtps: httpttt//ggtthbb..b.b http://sththuu...omm/1/1//m/ttt/

---

Algorithm 1: Successive Layer Training: w and W label the set of all layers' parameters.
Requires:
Number of rounds R, devices C, number of devices per round [etr), configurations Sn E [J, that satisfy constraint m, init. parameters W
Server:
foreach round =
do
Cir) + select |C(r)] random devices out of C
(((r ConfiuraaionSeleciion("""""
foreach device e ((r) in parallel do ()()) receive from server (n Deviceerainnnng (w(f), ((())
(ne)) server Vj Kj < <jsk) K}
upload m
end
3 (r+1)
rEC(r) w ( //averaging of trained layers
T00
W(++1) t w(r+l) //layers get embedded into server model end ConfigurationSSllction (W.r): r):
S + LoLoouTTTbble W) w.
w++W
//scaling down NN head based on configuration
Return S
Devicnng (w, S):
freeze w, for j E {0 < j S KF} according to S foreach local mini- batch b do
w + w - (V
end
Return w
behavior can be observed w.r.t. Floating Point Operations (FLOPs), where SLT requires 10x fewer operations reach a certain level of accuracy compared with FD and FedRolex. We study the performance of SLT in heterogencous settings. We show that devices with different memory constraints make meaningful contribution to the global model, significantly outperforming the state-of-the-ar techniques.
2 Methodology
2.1 Problem Statement and Setup
We consider synchronous cross-device FL setting, where we have one server and a set of devices E C as participants. There is given MI model topology F the FL server that is trained in distributed manner for rounds. Our goal is to maximize the accuracy of the model. Similar to FD)
and FedRolex, we assume that fixed number of devices |C(*) out of C participate a round r S R. All devices constrained in memory, and thus their training must not exceed this given memory constraint Mconstraint- In other terms, we assume that no participating device can train the server NN model end-to-end.
2.2 Successive Layer Training
The following describes methodology of Successive Layer Training for convolutional neural networks (CNNs). Firstly, we rewrite F such that it is the consecutive operations of K layers, where each layer is defined Jk. [1...., K]. Each layer f& has associated server parameters W,- We label convolution, followed by batch normalization and an activation function, layer. Similar we define a subset wi of the layer parameters (server) Wi that is scaled down using
g 20
wi W
Wk E B
We R PMM,
(I)

---

Fr
Fи
FF
Pr
Fи
W WKT
W
Kr+1 step n
W Kr+2
W
Ky+3
WKF
WKT
Wh
Ky+1
W
Kr+2
step
Visualization of SLT. With each step, KF and KT shifted by 1. Wi.... WKy denote the parameters that remain frozen during training, WK, denotes parameters of layer KT that are fully trained, while Whe Ky+1 W denote the parameters the scaled-down head using 8.
Figure 2:
where P& labels the layer's input dimension, M₁ labels the output dimension of the fully-sized server parameters, and s (0, 1] is a scaling factor (we omit the filter kernel dimensions for brevity). To obey the memory constraint on the participating devices, we split the NN into three consecutive parts. The first part of the NN contains layers that are already trained and remain frozen on the devices. The second part contains layers that are being fully trained on the devices. The last part represents the NN's head. To train the remaining layers within the memory budget, the head's parameters are scaled down. Throughout the FL training, we successively switch the training configuration, s.t., the part of the NN that being fully trained (s = 1) moves from the first layer to the last layer. Thereby, successively, the remaining parameters from the scaled-down head are added. At the same time, successively freeze more layers, starting with the first layer, to stay within the memory budget. We visualize the switching from training configuration to the next in Fig. 2 The parts of the NN that are frozen, trained, and represent the head are labeled FF, Fr, and FH- The resulting model that is trained on the devices can be described as F FF Fr FH:
The first part FF labels the part of the NN that is frozen and where the server pa- rameters W1,..., WKr do not get updated by the devices. Freezing the first part of the NN reduces the memory overhead during training, as in the forward pass, activa- tions do not have to be stored for frozen layers. The frozen part of the NN is defined as Fp:
OkE (k:Cck fk. where layers 1, 1,..., KF remain frozen.
The second part FT labels the part of the NN that is being fully trained by the devices. The parameters W Wx+t...... WKy get updated during training. This part is defined as Fr == layers KF I...., Kr are fully trained.
D Kr Kr
The last part FH describes the NN's head that is scaled down using 8, s.t. Fи (RKKKKKKK < where the scaled-down layers Kg + I....,A K are trained. The first layer of Fnr scales down the parameters to the width of the head s.t. WKy+1 W.. Kr+1 E R PR x|aMKy+11, All consecutive layers are scaled down using s, s.t. WKr W..s W E [ K- KT).
where WL. Kr+1
We define S as training configuration that obeys the given memory constraint, s.t. S {K Kr, 3} fully describes the mapping of layers fx, k E [I, K] into Fe. Fr, FH. and the head's 's scale factor s. Each has respective memory footprint during training. m memory(S) denotes the maximum memory that is utilized during training for a given configuration S. The maximum memory of configuration can be determined by measurements or by calculating the size of weights, gradients, and activations that have to be kept in memory (see Section 3.2
2.3 Configuration Selection
For each selected S, we aim fully utilize the available memory. We define n as a configuration step in E [1.... .......], .V1 where we add parameters to the training file..fil up the remaining parameters of a head's layer). These steps are distributed over total training rounds R Fig. 3}. We set Kr= = KF + 1, s.t. in each configuration step exactly one layer gets fully trained (filled up). We start with Kr = KT (consequently = Fir) to pre-train the head for certain number of rounds. After pre-training, increase KF by one and apply KT KF + (the first configuration has

---

pre-training
step 1
step
step .V (= 3)
[12: R
Figure 3: Visualization of the SLT training scheme with an exemplary 5 5-layer NN. The model is first pre-trained for rp rounds. Following that, the model is trained for vi - Tp. 12 Y1, and 72 rounds in configuration 1, 2, and ([= 3), respectively.
frozen layers, i.e., F Fr Fi) and continue training We switch to the successive configuration by increasing KF by one. Hence, for the training configuration at step n, we have KF = n 1 and Kr n, with 8n selected as follows:
max 8nz S.I. memory(S,) S coonstraint A 8n s, (n NJ.
(2)
Equation 2 ensures that each configuration obeys the constraint Mecomitraitt The second constraint in Eq. (2) enforces that 8, can only grow with increasing n to ensure that parameters of the head are only added throughout the training but not removed. We provide a justification for maximizing s instead of Fr by performing an ablation study in Appendix B The configuration selection is performed offline. Lastly, we define the last step N where a given memory constraint (memory (Sx)) allows for 1. If this step is reached,
train with Sw for all remaining rounds since the memory budget allows to fully train all remaining parameters at once. We provide a visualization of the training process in Fig. 3 and outline SLT in Algorithm
3 Experimental Evaluation
3.1 Experimental Setting and Hyperparamters
We evaluate SLT in an FL setting using Py (21). where we distribute a share from the datasets
CIFAR1O, CIFARIOU 100 (22), FEMNIST from the Leaf (3 benchmark, and TinyImageNet [24] to
each device с E C, s.t. each device e has a local dataset D. of the same size. In each round r, a subset of devices ((r) is selected. We train with the optimizer stochastic gradient descent (SGD)
with momentum of 0.9, an initial learning rate of 1 = 0.1, and apply cosine annealing to r = 0.01
and weight decay of 1.0 10-5 We evaluate the vision models ResNet20, ResNet44 (25 and DenseNet40 (26). For each experiment, we report the average accuracy and standard deviation of 3 independent seeds after R. rounds of FL training. For CIFARIO, CIFAR and TinyImageNet, evaluate scenario with |C] 100 devices, where each round ((2) 10 devices actively participating. For FEMNIST, we evaluate with |C| 3550 and [[ 35. We train for R = 2500 rounds for CIFAR10, CIFAR100, 100, and TinylmageNet, and R = 1000 for FEMNIST. In each round r. each participating device iterates once over its local dataset. We apply standard image augmentation techniques like random cropping and horizontal and vertical flips to all datasets (horizontal and vertical flips are omitted for FEMNIST). An input resolution of 32 32 is used for CIFAR and FEMNIST (up-scaled from 28 x 28) and 3 x 64 x 64 for Tiny ImageNet. We use batch size 32 and perform 363 experiments in total, with an average run-time of 6 h on an NVIDIA Tesla V100.
Comparison with the state of the art: We compare SLT against several baselines. We introduce ((r) as the set of indices of the output dimension of layer k E [1..... ......K]] K] where r E [1...., R denotes the rounds. Consequently, for full-sized NNs [r(r) = M₁- The subset for training is scaled down by building dense matrix using the indices from ( s.t. the scaled- parameters
Wg E R xP& [M]] The consecutive layer's input dimension indices are equal to the last layer's output indices. The first layer's 's input dimension is not scaled to feed all color channels into the NN.
Small model: Devices train submodel where all filters per layer scaled down by s, all devices can train the submodel. The remaining filters are trained end-to-end throughout the rounds.
We discuss in the evaluation section how to decide the number of rounds a certain configuration should be trained before switching the next configuration.
5

---

The same submodel is used for evaluation. The indices of the output dimension of layer ls are selected using _/ Ix = {i: : 0 S [sMaJ}
FedRolex [20]: FedRolex creates a submodel by sealing the numbers of filters using s. Each device trains the same continuous block of indices. The filter indices trained on the devices are shifted in rolling window fashion every round. The server averages the trained block and evaluates using the full server model (s 1). The indices
((r) are selected with f mod M2 using
T = + + [ - -1}
if + [sMa] S Mr
(3)
Mr 1) U (O...., f [sMM] - M otherwise
FD 19: FD creates submodel by scaling down the number of filters using s. The indices of the filters are randomly sampled per device per round on the server. Hence, the indices f((r:)) of device e of round r a round-based per-device random selection of [sM&]] indices out of all Mr indices. The server aggregates the device-specific submodels after training and evaluates the full model (s = 1).
3.2 Memory Footprint during Training
The high memory requirements during training can be split into three groups: Firstly, the weights of the NN have to be stored in memory. This is required both for the forward pass and the backward pass. Secondly, for the calculated gradients in the backward pass, the activation maps of the respective layers have to be kept in memory. Lastly, the calculated gradients have to be stored in memory. In state-of-the-ar CNNs, the size of the activation map makes up for most of the memory requirements, while the size of the weights only plays a minor role, For ResNet44 and DenseNet40,, we measure that activations make up for 99% of the required memory for training, while gradients and parameters account for the remaining 1%. Consequently, the required memory linearly reduces with s for FD and FedRolex, as the number of layer's output channels [s.M determines the activation map's 's size. Similarly, for SLT, we measure the maximum amount of memory that is required during training by counting the size of the activation maps, as well as the loaded weights and gradients in training. For frozen layers, it is only required to load the parameters in memory, while no activation maps and gradients have be stored. For the fully trained layer Kr, it is required to store the layer's full parameters w as well as the full-size activation map and gradients in memory. For all other layers (NN head), memory scales linearly with s. We provide implementation details in Appendix
We evaluate memory constraints that are given by scaling down in FedRolex and FD by SFD/FedRolex E [0.125, 0.25, 0.5, 1.0] for experiments with ResNet and [0.33, 0.66, 1.0] for DenseNe In SLT, for given SoDe/ we adjust Sn for each step n in the following way
(4) to ensure that our technique obeys the same constraint the baselines. If SFD/FodRolex 1.0, all algorithms coincide with vanilla Federated Averaging (FedAvg) using the full server model.
memory(syDieedtttt) max 8n, s.t. memory(5. < memory (SFD/TedRoles A 8n S 83
Vj E [n
NJ.
We distribute the required steps N over the total rounds R. s.t. all parameters receive sufficient training. Specifically, we distribute the rounds based on the share of parameters that are added to the training within a configuration n. We calculate the number of all trained parameters Q by using K Kr. and 8 s.t.
E + PK PKr+ilsMKrr [sM + EisPa](MMll (5) ke(k:KppckKkt
Q(Ke,
RE(k:Ky+1<kSk)
and use Q to calculate the share of rounds R, for a step n. The share of rounds for pretraining is calculated using Rpretraining R Q10.0.s)
Q(0.0.1) For step 1, R1 R Q(0.1.s) Q(00011)
Repetraining. For all steps
1, we calculate the rounds using
Rs R Q(n 1, 8w) - Q(n 2, 1, 1,8211)
(6)
Q,0,0,1)
Lastly, the switching point for pretraining is "prettaining Rpretraiing and rn R Qmn-1.n..)) for all steps n.
Q(0,0.1)
For DenseNet, SLT only enables a reduction of 3x, as in DenseNet, specific layers have significantly larger sized feature map than others, which limits technique's effectiveness w.r.t memory reduction.

---

Table 1: Results for iid experiments with ResNet and DenseNet using CIFARIO, FEMNIST, CI-
FAR 100, and TinylmageNet. Accuracy in after R rounds training is given.
Setting
ResNet20CIFAR10
ReSNet2WEMMNIST
0.125
0.25
0.5
1.0
0.125
0.25
0.5
1.0
SLT (ours)
74.1=-0.8
83.4+00.2
85.2++0.6
84,4++0.3
85,8+0.1
86,9+0.0
Small model
6..3110.3
78.2.00.
84.6.10.4
82.3::0.4
85.5
86.910.00
87.5+006
87.6++0.0
FedRolex 20
11.7+
61.0+:0.5
80.6+0066
42.1+6600
71.4:2211
83.0+0.1
19.0+1.4
36.721.8
71.6+0.4
38.943.7
70.4+0022
83.440.3
Setting
DenseNer4//IIAAR000
ResNet4/TinylmageNe
0.33
0.66
1.0
0.125
0.25
1.0
SLT (ours)
51.1=0.44
53.3+0.6
33,5++0.1
40.3:1..5
42.3++0.2
Small model
43.91
55.910.11
19.8.10.3
30.6.10.3
40.2.1.0.3
60.210..5
45.210.1
FedRolex 20
22.2+-0.3
46.7+0.1
6.9+0.2
19.8+0.8
33.1+0022
13.5+0.5
41.911.5
0.94:0.0
7.1+0.1
25.940..6
Preliminary experiments have shown that this mapping scheme outperforms other techniques, like an equal distribution of rounds to all configurations, and enables SLT to converge as fast as a small model while reaching significantly higher final accuracy (we provide further results in Appendix [
The mapping of steps N to rounds R does not rely private data (or any data) and can be stored in look-up table prior the training. A visualization of SLT is given in Fig. We provide the number of steps N for different NN architectures and constraints in Appendix
3.3 Experimental Results
lid results: For the iid case, results are given in Table We observe that SLT reaches significantly higher accuracy for ResNet20 and CIFARIO 10 for all evaluated constraints, outperforming small model baseline by up 7.8 p.p. and state of the art by 52.4 p.p.. The results with FEMNIST show that a small model baseline already provides sufficient capacity for the dataset since only a few percentage points separate SFD/FedRolxx 0.125 from the full model (i.e., when SFD/FedRolex 1). Hence, SLT can only provide minor benefit over small model baseline. The contrary can be observed for CIFAR 100 and TinylmageNet, where using small model (SrDiTedRoles 0.125) loses up to 25.4 p.p. to the full model. Additionally, it can be observed that for low memory constraints, FD and FedRolex fail to learn a useful representation at all. SLT improves upon small model by up to 13.7 p.p. and up 26.6 p.p. compared to state of the art.
Non-iid results: Typically, data in FL is not distributed in iid fashion but rather non-iid. We repeat the experiments shown in T but distribute the data on the devices in a non-iid fashion. Similar [27]. we apply a Dirichlet distribution, where the rate of non-iid-ness can be varied using a. For all experiments, we set a 0.1. We observe from Table 2 that the small model baselines in the case of CIFARIO and FEMNIST lose accuracy compared the full model. Hence, the gain of SLT compared to a small model baseline increases. The results for CIFAR 100 and TinylmageNer show proportional drop in accuracy for all algorithms. However, SLT still outperforms other techniques by a large margin. Note that we could apply common non-iid mitigation techniques like FedProx [1] on top of SLT to further limit the drop accuracy.
For additional experimental results, we refer the readers to Appendix D
Communication, computation, and convergence speed: We evaluate technique w.r.t. the communication overhead of the distributed training and the number of computations devices have to perform (FLOPs). Specifically, and performed FL.OPs. We show the results in Fig. 4 We observe that our technique converges fast, similarly small model, while reaching higher final accuracy. Compared FD and FedRolex, our technique requires significantly less communication to reach the same level of accuracy. Similar behavior can be observed w.r.t. FLOPs.
evaluate the gain in accuracy
required transmitted data
3.4 Heterogeneous Memory Constraints
Memory constraints devices can be heterogeneous. evaluate SL.T' such scenarios and compare it against the start of the art. We evaluate with different resource levels and split the available constraint
7

---

Table 2: Results for non-iid experiments with ResNet and DenseNet using CIFARIO, FEMNIST, CIFAR100, and TinylmageNet. Accuracy in % after R rounds of training is given.
Setting
ReSRR100CIAA11
ReSNet2VVEMNIST
0.125
0.25
0.5
1.0
0.125
0.25
0.5
1.0
SLT (ours)
52.4+-0.9
69.6+:.66
75.5+1133
81,2++1.
83.0+2200
8388+199
Small model
44.7+1122
63.110.7
73.6::0.6
79.6:10.88
82.911.1
83.3::2,4
80.5+1.3
84.0+
FedRolex 20
15.0
19.8
48.3+:2.9
39.4++2.00
59.3:22.1
78.5+0..5
11.3+0.9
10.74:0.
34.945.7
15.948.2
51.0+1.2
79.7
Setting
DenseNe44//IFARI000
ResNet4/TinylmageNe
0.33
0.66
1.0
0.125
0.25
1.0
SLT (ours)
45.9::1.4
48.4+0.5
28.5++1.2
35.1=1111
36.1=00.2
Small model
40.5.112
$1.8.0.2
16.911.2
25.3:10.5
34.2.1...4
55.810.5
39.010.8
FedRolex
20
20.0++0.3
42.9+0.4
1.5++.5
12.7+
26.1
7.641.11
36.9+1.0
0.44:.1
0.6:0..0
20.0
CIFAR
Thy Imygeaggeeet
TinyImageNa
Ne40eNet40
ResNer20
ResNer44
ResNe144
0.33
0.25
SybitwfRets = 0.125
0.25
Small model
SLT (ours)
40 [i] Eatanoo
80
60
30
30
10
20
FedRolex
20
20
40
20
FD
20
10
10
10 3
10
10
10
10
10
10
10
10
10
10a
Upload [GB]
Upload [GB]
Upload [GB]
Upload [GB]
80
30
40
40
60
20
40
20
20
20
10
10a
10a
10
10a
10
10
10a
10
10
10a 10 10
GFLOPs
GFLOPS
GFLOPS
GFLOPS
Figure 4: Maximum reached accuracy in % over data upload and performed FLOPs for CIFAR1O, CIFAR100, and IinylmageNet using DenseNet40, ResNet20, and ResNet44 in an iid training case.
levels equally upon the devices, i.e., when constraints of @FD/FedRolex = [0.125, 0.25] are given, 50%
of the devices train with SFD/FedRolex 0.125 while the remaining 50% use SFD/FedRolex 0.25. SLT supports heterogeneous constraints through the following mechanism: Firstly, devices with highest constraint perform training done in the homogeneous case, outlined in Algorithm uusing the head's ' scale factor as described in Eq. (2). Devices that are less constrained use the same scale factor sn per configuration to ensure that all devices train the same number of parameters within a layer. To utilize the remaining available memory, less constrained devices freeze fewer layers, therefore, train more layers at full width. For given Kr and 8n of configuration S,, the remaining memory of less constrained devices is utilized by minimizing K3, s.t.
min KF, memory(S.) < memory (wrtorryySSpFFre SpD//edRolex)
In addition to FD and FedRolex, we evaluate HeteroFL [18 and FjORD [19]. Both require that some devices are capable of training the full NN end-to-end, otherwise, some parameters do not receive any updates. In cases where no device can train the server model end-to-end, we reduce the size of the server model such that at least one participating device can fully train the model.
Small model: All devices train a small model regardless of their constraints. Scale factor s set to the minimum participating device supports.
FedRolex 20): Similar to the homogeneous case, FedRolex uses a rolling window (Eq. heterogeneous cases, devices use constraint-specific scale se, to adjust the number of filters s Mr
In
8

---

Table FL with heterogeneous constraints. Accuracy in % after R rounds of training is given.
Setting
DenseNatd0/CIFAR1000
ResNac4/TinyymageeNe
SyD/FedRolex
[0.33, 0.66]
[0.33, 0.66, 1.0]
[0.125, 0.25]
[0.125. 0.25,
[0. 0.25, 0.5, 1.0]
SLT (ours)
46.4+:2.0
4..3=118
3..3+1.2
33,0=+0.5
35.9=0.4
Small model
1.2
1..
16.9=0.2
16,9:00.2
16.9=0.2
FedRolex 20
33.2++0.4
1339=113
5,4
13.8+11.3
23.6==0.7
FD 9
21.2++0.6
38,1=004
0.5+
0.6+0.1
10.6=
HeteroFL 18
1..2
4..8=0.5
20.7+0.77
24.1::02
23.3=0.55
FjORD
19
38.710.4
36,910.4
22.4110
25.3:00.
27.5
FD 9: Although Caldas el al. [9] do not specifically evaluate heterogeneous devices, heterogeneity can supported straightforwardly by using constraint-specifie sa for scaling down the NN. This extension of FD is also evaluated in FedRolex and FjORD.
HeteroFL 18] : In HeteroFL, devices use the same subset throughout the training. To support heterogeneity, devices use resource-specifie scaling factor Se, s.t. for each se the indices are selected using T (r.e) = {i 0 < Mx
FjORD 19: FjORD uses the same indices as HeteroFI., but each device switches between constraint- specific subsets that satisfy the device constraints within local epoch on mini-batch level.
Heterogeneity results: We repeat the experimental setup as presented in Table for TinyImageNet and CIFARI00, but enforce varying device constraints in the experiments (see Table 3). We observe that SLT outperforms others in all evaluated scenarios. FedRolex can improve upon a small model in some settings, but this is not the case with FD. For FjORD and HeteroFL, we observe that both outperform the small model baseline. Yet, in some cases, both HeteroFL and FjORD have lower accuracy when utilizing more constraint levels. For HeteroFl., it can be observed that using ResNet44 with constraint levels reaches lower accuracy than with 3 levels (despite the fact that all devices have higher average resources E (SDD//ddRles)) of az 0.47 in the case of [0. (2.125, 0.25, 0.5, 1.0] instead of xs 0.29 in the case [0.125, 0.25, 0.5)). The same can be observed for FjORD with DenseNet40. As both techniques, in principle, use the same subset mechanism as FedRolex and FD, we think that both suffer from supporting more constraint levels that cause less co-adaptation between NN filters.
4 Related Work
We cover related work that studies similar problems or employs similar techniques.
Resource constraints in FL: Most works on resource-constrained FL target communication. Specifi- cally, the use of quantization and compression in communication p
and sketched updates [IO]
have been proposed to lower the communication burden. Chen et al. (28 propose adaptive parameter freezing they discover that parameters stabilize during training and do not have be transferred to the server. Another branch of work focuses on reducing communication. computation, and memory requirements by employing only subset of the full NN on devices. Caldas er al. 9 introduce FD, mechanism that creates a device-specific subset by randomly selecting a subset of CNN filters. Diao er al. [88 introduce HeteroFL, which allows for heterogeneous constraints by employing fixed subsets of different sizes to the NN and aggregating them on the server. Horvath el al. [19 introduce similar technique (FjORD), with the main difference that in FjORD, each device trains every available subset within its capabilities. Rapp el al. (29 propose DISTREAL. technique that uses varying subsets on a mini-batch level such that devices finish their update on time despite having intra-round changing resources. FedRolex (20) supports heterogeneity similar to FjORD and HeteroFL but allows for server NN models that are outside of the capabilities of all devices. This is enabled by not training a fixed subset of the NN parameters but by training a rolling window of all parameters that is shifted on a round basis. Beyond subsets, the use of low-rank factorization 30 31 has been proposed to train NN models. Lastly, Qui et al. [32) propose sparse convolutions lower the resource requirements for training but require special hardware for sparse computations to realize the gains.
Layer-wise model training: Layer-wise model training has been proposed in centralized training of CNNs as an alternative to training with end-to-end backpropagation of the error. Hettinger et al. [33] introduced technique that adds CNN layers one time during training using auxiliary

---

heads for classification while freezing early layers. Similar techniques have also been employed for unsupervised learning, where representations are trained with contrastive techniques without Recently, the concept of progressive model growth has also been proposed for FL.: Wang et [36] propose ProgFed, where they discover that by progressively adding CNN layers to the NN while using an auxiliary head, the FI. training converges faster and required less communication to reach the same accuracy as an end-to-end baseline. Similarly, Kundu et al. (37 propose a technique that grows the model depending on the complexity of the data to reach a high accuracy if the NN capacity is not sufficient for the problem. Importantly, both techniques only focus on increasing the convergence speed. Hence, they consider communication and computation overhead but not the problem of constrained memory on edge devices, nor do they support heterogeneous devices. In both techniques, eventually, all devices have to train the full-size NN and, consequently, have to have the memory resources available for that.
requiring end-to-end gradient propagation R4 35
Memory-efficient training: Several techniques have been proposed to train an MI. model in memory-efficien way. Kirisame et al. (38 present Dynamic Tensor Rematerialization that allows recomputing activation maps on the fly. Similarly, encoding and compression schemes [39 40) have been proposed to lower the size of the activation maps during training. Techniques like that trade memory for computation, and some lower the accuracy by using approximation or lossy compression.
Importantly, these techniques are orthogonal to SLT. FD, FedRolex, HeteroFL, and FjORD.
5 Conclusion
We proposed SLT that is able to reduce the memory requirements for training on devices, efficiently aggregating computation capacity and learning from all available data. Through extensive evaluation, we show that gains in final accuracy as well as the faster convergence speed (compared with state of the art) robust throughout different datasets, data distribution, and NN topologies.
Limitations: We observe that SLT is most effective if the used NN architecture is deep (i.e., has many layers), as the cost of filling up single layer becomes less significant. Also, SLT is less effective if the size of the activation map is strongly unevenly distributed throughout the layers (DenseNet), as has to adapt to the layer with the highest memory requirements when filled up. Besides, we applied SLT to CNN topologies only. Finally, we mainly focused on memory as a hard constraini [41] for training. We show that communication and FLOP efficiency are significantly higher than in the state of the art, but we did not consider per-round communication or FLOP constraints. For future work, we want to extend our study to other topologies, such as transformers, and employ neural architecture search (NAS) techniques to find NN configurations that reach the highest accuracy when trained in FL with SLT in heterogeneous environments.
Our solution could help reduce biases FL systems, improving fairness. For instance, by including users with low-end smartphones in the learning process, it provides these users (who perhaps cannot afford high-end devices) with better experiences as the model is going to be trained over their data, too. It could also reduce the cost of deployment of distributed IoT systems (e.g., sensor networks), as they can be implemented with low-cost devices (or a mixture of low and high-cost devices), enabling, e.g., deployment of larger and more tine-rrined monitoring systems. On the negative side, distributing learning over low-end devices that are not particularly designed for training tasks increase the overall energy consumption of the system. This is important issue that should be studied in more detail.
Broader impact:
Acknowledgments and Disclosure of Funding
This work was partially funded by the "Helmholtz Pilot Program for Core Informatics (kikit)" at Karlsruhe Institute of Technology. The authors acknowledge support by the state of Baden- Württemberg through bwHPC.
References
[1] Yiqiang Chen, Xin Qin, Jindong Wang, Chaohui Yu, and Wen Gao. Fedhealth: A federated transfer learning framework for wearable healthcare. IEFE Intelligent Systems, 35(4):83-95 2020.
10

---

[2] Binhang Yuan, Song Ge, and Wenhui Xing. A federated learning framework for healthcare iot devices. arXiv:2005.05000 005.00 2020.
[3] Bekir Sait Ciftler, Abdullatif Albaseer, Noureddine Lasla, and Mohamed Abdallah, Federated learning for localization: A privacy-preeeviing crowdsoureing method. 1.Xiv:2001019111 2020.
[4] Jason Posner, Lewis Tseng, Moayad Aloqaily, and Yaser Jararweh. Federated learning in vehicular networks: Opportunities and solutions. IEEE Network, 2021.
[5] Boyi Liu, Lujia Wang, and Ming Liu. Lifelong federated reinforcement learning: a learning architecture for navigation in cloud robotic systems. IEEE Robotics and Automation Letters, 4(4):4555 4562, 2019.
[6] Boyi Liu, Lujia Wang. Ming Liu, and Cheng-Zhong Xu. Federated imitation learning: A privacy considered imitation learning framework for cloud robotic systems with heterogeneous sensor data. arXiv: /909/ 00895, 2019.
[7] Farzad Samie, Vasileios Tsoutsouras, Lars Bauer, Sotirios Xydis, Dimitrios Soudris, and Jörg Henkel. Computation offloading and resource allocation for low-power iot edge devices. In 2016 IEEE 3rd world forum on interner of things (WF-loT), pages 7-12. IEEE, 2016.
[8] Jed Mills, Jia Hu, and Geyong Min. Communication--fficce federated learning for wireless edge intelligence in iot. IEEE Internet of Things Journal, 7(7):5986-599) 5994,,9944444 2019.
[9] Sebastian Caldas, Jakub Konečny, H Brendan McMahan, and Ameet Talwalkar. Expanding the reach of federated learning by reducing client resource requirements. 18/2.02// 2018.
[10] Daniel Rothchild. Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica. Vladimir Braverman, Joseph Gonzalez, and Raman Arora. Fetchsgd: Communication-effcico federated learning with sketching. In International Conference on Machine Learning, pages 8253-8265 PMLR, 2020.
[11] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, volume 2, pages 429 450, 2020.
[12] Y. Chen, Y. Ning, M. Slawski, and Rangwala. Asynchronous online federated learning for edge devices with non-iid data. In 2020 IEEE International Conference on Big Data (Big Data), pages 15 24, 2020.
[13] Cong Xie, Sanmi Koyejo, and Indranil Gupta. arXiv:1903.03994 2020.
Asynchronous federated optimization.
[14] Zheng Chai, Yujing Chen, Liang Zhao, Yue Cheng, and Huzefa Rangwala. Fedat: A cmmmunication-effficae federated learning method with asynchronous tiers under non-iid data. arXiv:2010.05958 2020.
[15] Zheng Chai, Ahsan Ali, Syed Zawad, Stacey Truex, Ali Anwar, Nathalie Baracaldo, Yi Zhou, Heiko Ludwig, Feng Yan, and Yue Cheng. Tifl: A tier- based federated learning system. In Proceedings of the 29th International Symposium on Performeffrmeeee Parallel and Distributed Computing, HPDC '20, page 125-136. New York, NY, USA, 2020. Association for Computing Machinery.
[16] Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Françoise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv: 18/ 0209903, 2018.
[17] Kiwan Maeng, Haiyu Lu, Luca Melis, John Nguyen, Mike Rabbat, and Carole-Jean Wu. Towards fair federated recommendation learning: Characterizing the inter-dependence of system and data heterogeneity. Proceedings of the 16th ACM Conference on Recommender Systems, pages 156-167, 2022.

---

[18] Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication effi- cient federated learning for heterogeneous clients. In International Conference on Learning Representations, 2020.
[19] Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis. Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 34:12876-12889, 2021.
[20] Samiul Alam, Luyang Liu, Ming Yan, and Mi Zhang. Fedrolex: Mloeel-heteoggenouu federated learning with rolling sub-model extraction. In Advances in Neural Information Processing Systems, volume 35, pages 29677-29690, 2022.
[21] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, pages 8024-8035, 2019.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
[23] Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný. H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv:1812.01097, 2019.
[24] Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge, 2015.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[26] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 2261-2269 2017.
[27] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv:/909. 06335, 2019.
[28] Chen Chen, Hong Xu, Wei Wang, Baochun Li, Bo Li, Li Chen, and Gong Zhang Communication-effiiieee federated learning with adaptive parameter freezing. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS), pages 1-11. IEEE,
2021.
[29] Martin Rapp, Ramin Khalili, Kilian Pfeiffer, and Jörg Henkel. Distreal: Distributed resource- aware learning in heterogeneous systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8062-8071, 8071, 2022.
[30] Dezhong Yao, Wanning Pan, Yao Wan, Hai Jin, and Lichao Sun. Fedhm: Efficient federated learning for heterogeneous models via low-rank factorization. arXiv:2111.4465: 2021.
[31] Yiqun Mei, Pengfei Guo, Mo Zhou, and Vishal Patel. Resource-adaptive federated learning with all-in-one neural composition. In Advances in Neural Information Processing Systems, volume 35, pages 4270-4284, 2022.
[32] Xinchi Qiu, Javier Fernandez- Pedro PB Gusmao, Yan Gao, Titouan Parcollet, and Nicholas Donald Lane. Zerofl: Efficient on-device training for federated learning with local sparsity. In International Conference on Learning Representations, 2022.
[33] Chris Hettinger, Tanner Christensen, Ben Ehlert, Jeffrey Humpherys, Tyler Jarvis, and Sean Wade. Forward thinking: Building and training neural networks one layer time. arXiv:1706.02480. 2017.
12

---

[34] Sindy Löwe, Peter O' C'Conor, and Bastiaan Veeling. Putting an end to end-to-end: Gradient- isolated learning of representations. Advances Neural Information Processing Systems, 01330099-051,,20
[35] Yuwen Xiong, Mengye Ren, and Raquel Urtasun. Loco: Local contrastive representation learning. Advances in neural information processing systems, 33:1 15:11142-11155 11113,, 2020.
[36] Hui-Po Wang, Sebastian Stich, Yang He, and Mario Fritz. Progfed: effective, communication, and computation efficient federated learning by progressive training. In International Conference on Machine Learning, pages 23034-2305$ PMLR, 2022.
[37] Amit Kumar Kundu and Joseph Jaja. Fednet2net: Saving communication and computations in federated learning with model growing. In Artificial Neural Networks and Machine Learning- ICANN 2022: 31st International Conference on Artificial Neural Networks, Bristol, UK, Septem- ber 6-9, 2022, Proceedings; Part IV, pages 236-247. Springer, 2022.
[38] Marisa Kirisame, Steven Lyubomirsky, Altan Haan, Jennifer Brennan, Mike He, Jared Roesch, Tianqi Chen, and Zachary Tatlock. Dynamic tensor rematerialization. In International Confer- ence on Learning Representations, 2021.
[39] Animesh Jain, Amar Phanishayee, Jason Mars, Lingjia Tang, and Gennady Pekhimenko. Gist: Efficient data encoding for deep neural network training. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), pages 776-789. IEEE, 2018.
[40] Georgios Georgiadis. Accelerating convolutional neural networks via activation map compres- sion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[41] Kilian Pfeiffer, Martin Rapp. Ramin Khalili, and Jörg Henkel. Federated learning for com- putationally constrained heterogeneous devices: survey. ACM Comput. Surv., 55(14s), jul 2023.
[42] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929 19519 2014.
[43] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. lece, 2009.
13

---

A Co-Adaptation in Subset-based FL
Federated Dropout originally inspired by regular dropout 42 a regularization technique that constrains the capacity of a large NN model by randomly dropping parameters from the training, thereby limiting oo-adptationn among parameters of the model. This essential to improve the accuracy and reduce the over-fitting of parameters, as shown in various studies. FD and FedRolex adopt the dropout technique, removing CNN's filters in a round-based manner. These techniques, however, exercise dropout to its extreme, dropping large part of filters so that not enough co- adaptation between filters remains. In particular, the gradients for the subset of parameters trained on device are calculated without consideration of the error of the remaining parameters that reside the server. These subsets are randomly changing over time and devices, reducing the co-adaptation of this distributed training process. Add to these the fact that the data is also distributed over devices, so applying such a random scheme significantly decreases the chance that a subset of parameters is being trained together
sizable proportion of the data.
To further study the effects of co-adaptation on the reachable accuracy and the differences between FD and FedRolex, we run the following experiment, using CIFARIO with ResNet20 and SFD/FedRolex 0.25:
We modify FD s.t. all devices train the same random subset per round, i.c., the same indices ((ne) = 7(r) per round (index & is omitted for simplicity).
We limit the randomness, where at each round, we arbitrarily select ((r) out of a set I = {I,..... I of randomly initialized subsets that generated once prior to training.
80
Small model
accuracy [%]
FedRolex (20
FD g with limited randomness
60
400
200
400
[
Figure 5: FD with limited randomness using CIFAR1O with ResNet20 and SyD/eeRRoles 0.25.
We vary the randomness by varying the number of sampled subsets in s.t. [Z] E [0, 500]. Thereby, the probability of a specific subset being selected is = jzj Recall that all the devices train the same submodel in round, and there only [Z] submodels that would be trained by devices training period. Evaluation is done with the full server model. Results are shown in Fig. 5
the
We observe the following effects: 1)
The final accuracy drops proportionally to p. 2) In the case of [Z] = 1, FD behaves similarly to the small model baseline, as always the same subset is used for training. We also observe that remaining untrained filters have minor effect on the accuracy when compared with small model. However, because of these untrained parameters, the model fails to reach higher accuracies with SLT (see Section 3). 3) The accuracy drops with introducing more randomness to the training process (i.e., increasing [T. This is as co-adaptation among parameters of the model reduces as we increase the randomness. The rolling window approach of FedRolex is a special case of FD with limited randomness (i.e., [ = 5 in this experiment).
B Ablation study maximizing over Fr
To justify our design choice in Section 2 to maximize 8, for all steps n, we conduct an ablation study, where we study the best trade-off between s and Fr. In particular, instead of maximizing we only use fractions of the maximized s, labeled Sablation We evaluate different values for Sablation i.c. en E (0, 1]. When only a fraction of the maximized sn is used in a step, the remaining memory can be used to increase the size of Fr. We conduct with CIFAR Net2 and TinyIm- Nee, where all the hyper-parameters are kept the same as in Section 3.T (except the changes in sn and Fr). The final accuracy of SLT is displayed in Fig. 6 For each run, we depict the
14

---

0.25
TnnylmmgeNN//RsseeeH4//rrrrrce
0.125
35
accuracy [%]
80
30
75
25
70
20
15
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Platstion
Figure 6: Trade-off between maximizing over Fr. The results show that maximizing gives the highest accuracy.
(Satamm
1)
Table 4: Steps N in SLT for different constraints and NN models.
Constraint ResNet20
ResNet44
DenseNet40
0.66
0.33
15
0.25
14
26
0.125
16
14
average accuracy and standard deviation for three seeds. The results show that by maximizing in favor of Pr, SLT reaches higher final accuracy.
C Mapping of steps N to rounds R
Generally, any layer that trained SLT should receive sufficient amount of training extract useful features for downstream layers, but at the same time, it should not overfit in the current configuration. The mapping of rounds R steps N in SLT is done proportionally of added (previously untrained) parameters to the training. Since N depends on how many steps are required until 1, the number of steps depends on the constraint level. Table we list N for all experiments and constraints. Fig. visualizes this mapping for DenseNet40, ResNet20, and ResNet44, where SLT''s accuracy over rounds is displayed in green while steps over rounds are displayed in black. This mapping scheme has key advantages over other techniques. Most importantly, depends only on the NN structure and not on the data available on the devices. Hence, it can be calculated offline prior to the training.
the amount
We compare our mapping scheme with other mapping schemes, one offline and one online:
Equal distribution: In this scheme, we equally distribute the rounds to the steps, i.c., R,,
R
In this scheme, decide online, based the test accuracy, when to switch. the test accuracy on the server for number of FL rounds does not improve, the mapping switches to the next configuration. The number of rounds is usually referred to as patience. We evaluate with patience 5, 15, and 25.
Early stopping:
To compare the mapping schemes, we run experiments with CIFAR10 and ResNet20, where, except for the mapping scheme, all hyperparameters are kept the same (as presented in Section 3)). We can observe from the results in Fig. 8 that our mapping scheme outperforms the others with respect to the final accuracy and convergence. Even though the early-stopping based technique with patience increases n more aggressively, it does not result in faster convergence or higher final accuracies.
D Miscellaneous Experiments
To evaluate how SLT and baselines perform with more complex datasets and deeper NNs, we evaluate it with the full ImageNet [43] dataset (1.28M images, 1K classes). However, downscale the
15

---

CIFARI00
CIFARIO
TinylmageNet
DenseNet40 SFD/FedRolex
0.33
ResNer20 SFD/FedRolex
0.25
ResNet44
SFIMFedRolex = 0.125
15
80
10
30
30
aceuracy [%)
40
10
accuracy [%)]
accuracy [%6]
steps
steps
20
20 step
40
20
10
10
20
0
0
0
1,000 2,000
1,000 2,000
1,000 2,000 Rounds
Rounds
Rounds
Figure 7: Accuracy over rounds (green) and steps over rounds (black) in SLT for CI- FAR100/DenseNet40, CIFARI0/ResNet20,, and TinylmageNet with ResNet44.
15
SLT
accuracy [%)]
60
SLT (equal share)
steps
10
SLT (early stopping 5)
40
SLT (early stopping 15)
5
SLT (early stopping 25)
20
0
1,000
2,000
1,000
2,000
Rounds
Rounds
Figure 8: Different strategies for mapping N to R using CIFARIO with ResNet20 and SFD/FedRole 0.25. Accuracy over rounds is displayed on the left. Steps over rounds are displayed on the right.
images to 3 x 64 x 64 pixels to reduce the complexity of the evaluation. To account for the larger dataset, increase the number of devices to |C| 500 and the number of rounds to R 4000. All remaining hyperparameters are kept the same (Section 3}. We use ResNet56 to account for the more complex dataset. Results for iid and non-iid data provided in Table We observe that the general trend of Tables [ remains the same: SLT outperforms the state of the art and the small model baseline with large margins.
E Training Memory Measurements in PyTTrch
We measure the maximum memory requirement for the evaluated NN models ResNet and DenseNet using PyTorch 1.10. Specifically, we measure the size of the activations, gradients, and weights. These memory measurements are done offline (prior to training) and do not require any data.
Measurement weights: To measure the size of the weights, we sum up all tensors that are present in the NN's state_dice
Table 5: Results for iid and non-iid experiments with ResNet56 using Imagenet (64 x 64) are given. Accuracy % after 4000 rounds of training is given.
Setting
ResNet56//mngeNe//id
ResVet55/lmageeeeeoonn-ii
0.125
0.25
1.0
0.125
0.25
1.0
SLT (ours)
24.2-10.2
31.2-10.3
34.6:10.1
21.7-10.9
29.7:10.3
31.8-10.4
Small model
8.9.1..2
18.4.00.
30.310.1
8.4.1..3
16.2.10.2
27.3:10.3
41.6.10.5
38.7.10.3
FedRolex 20
3.4--0.2
114+-0.3
21.3+0.5
2.8+-..1
10.2+:0.9
18440..
0.3:1.22
6.240.22
16.810.5
0.14:..0
0.1::0.0
15.740.4
16

---

Measurement of activations and gradients: To measure the size of the activations that have be kept in memory, as well as the gradients, we apply backward_hoole PyTorch modules in an NN. Specifically, we add these hooks to Conv2d, BatchNorm2d, ReLU, Linear, and Add operations. If hook attached to a module called, we add the respective size of the activation map and the size of the calculated gradient to global variable to add up all required activations and gradients.
to all relevant
17


Reviews:
[{"review_id": "JZDlrNyrdv", "reviewer": "Reviewer_cUHB", "summary": "This paper proposed a new training method for neural networks in the federated setting, called successive layer training. This method tries to train NN layer by layer by freezing the trained layer and scaling down the NN heads. The authors have a clear presentation for this method and propose detailed experiments to show the performance. ", "questions": "Although this strategy here is proposed under the background of FL, it is a general approach suitable for any distributed environment. So is it effective for applying the method in standard distributed training, while communication is the bottleneck.  Meanwhile,  is it possible to apply this strategy to attain the accuracy of full training? ", "limitations": "Some suggestions are as follows:\n\n1. Give some explanation for this method\n2. Do adequate experiments on standard nets and datasets \n3. Study this training strategy as a general approach first and then apply it to FL", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The experimental results show good performance when compared with other algorithms for this memory-constrained setting. It is exciting to see that such an (approximate) layer-by-layer training strategy can have an acceptable accuracy. The configuration for this setting is also flexible when adapting to environments with different resource constrained. ", "weaknesses": "The method itself is somewhat straightforward and the paper makes no explanation about why this strategy can work. Meanwhile, the experiments were only done on some toy neural networks and thus were not very representative. At least testing on Resnet50 on Imagenet is necessary.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We like to thank the reviewer for the fair and constructive feedback on our manuscript.\n\n# Scope of the work\nOur work is motivated by the observation that the current state of the art in memory-aware-FL through scaling of the NN does not work well. We think that the bad performance is caused by the fact that gradients of the subset of a layer are calculated (for a local epoch) without consideration of the parameters that reside on the full server NN (as also discussed in Appendix A). We think this problem is tightly coupled to the federated averaging mechanism which is used in federated learning. Hence, as the state of the art does, we target federated learning environments. \n\nWe show in Figure 4 in the paper that SLT requires significantly less communication to reach the same level of accuracy as the state of the art, so SLT could be also effective in distributed training settings where the communication is the bottleneck. However, this should be studied more in detail and is thus delegated to future work.   \n\n# ResNet50/ImageNet\nWe think that currently for devices in federated learning, such as smartphones, IoT devices, and sensors, it is not feasible to train large models like ResNet50 on ImageNet (full image resolution). We want to add that we also lack the capabilities hardware-wise to simulate ResNet50/ImageNet in a federated environment as a simulation of FL takes considerably more time than centralized training. \n\nThrough our study, we observed that SLT benefits from two factors: a) a larger capacity gap between a small model ($0.125\\times$) and the full model ($1.0\\times$); and b) deeper NNs, as the memory cost of *filling up* a single layer reduces, thus allows for larger values of $s$ throughout the training. We believe that this effect would also transfer to larger NNs and more complex problems like ResNet50 and ImageNet. \n\nTo provide further evidence for that we evaluated a downscaled version of ImageNet (64 by 64 pixels) with ResNet56 (one of the cheaper CIFAR10 adapted family of ResNet models), where with 500 devices SLT reaches $22.0\\\\%$ of Top-1 accuracy, while the small model baseline only reaches $8.2\\\\%$ Top-1. It can be seen that compared to TinyImageNet, this is already a larger gap. We are confident that the FedRolex and FjORD would perform worse than a small model, as it is the case for all other experiments."}, {"review_id": "LCj41r3APn", "reviewer": "Reviewer_iW6b", "summary": "In this paper, authors study the case where the edge device is not capable of holding the training of the whole network due to limited memory. Authors find that it is necessary to train a whole network rather than train a submodule as previous methods do. Motivated by this observation, authors propose to first scale down the intermediate size of the network and then expanding it as its precedent layers being fully trained. Authors provided experimental results and those results successfully proved that SLT outperforms other algorithms in this scenario.", "questions": "Please refer to my questions in the weakness part.", "limitations": "Authors have clarified their limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The observation that it is necessary to train a full network, rather than a submodule, sounds reasonable to me. Moreover, the SLT method proposed looks new to me. This idea could be helpful for future studies in memory-constrained FL training.", "weaknesses": "The paper is not clearly written in some parts, such as Section 2.2. My major concerns come from two parts:\n1) Why SLT can achieve almost the same result w.r.t. the ''small model'' method when the model can accommodate the edge device, as shown in Figure 4? To me, the shallower layers are only trained when the deeper layers get forward partially in the early stage, and they are kept frozen when training the deeper layers. This means the shallower layers are not fully trained with the correct gradient that comes from the whole model, which means SLT is not equivalent to the full training algorithm and it shall get a worse performance than the ''small model'' method. Therefore I doubt why it can get a matched accuracy as shown in Figure4.\n2) Actually the footprint of activations can be alleviated by checkpointing or offload, please compare SLT with these methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We like to thank the reviewer for the fair and constructive feedback on our manuscript.\n\n# SLT's improvements over a \"small model\" baseline\n* The observation is correct that parts of the model are never trained end-to-end in SLT. However, SLT pretrains a subset of all layers end-to-end in step $0$ (equal to an affordable small model), where no layer is frozen $K_T=0,K_F=0$. After this pretraining phase, gradually, step by step, a layer is *filled up* while shallower layers get frozen. This way shallower layers are trained to still produce useful features for the downstream head. Thereby, SLT reaches higher accuracies than a small model that has the same memory footprint in training. This is especially the case if the small model does not have sufficient capacity for the given dataset. SLT allows training models with larger capacities. Despite the error introduced by our training scheme being not end-to-end, the gain in capacity outweighs this error as can be seen in Figure 4 and Table 1. At the same time, SLT is not able to reach the accuracy of the full NN, hence can not compete against an end-to-end trained full NN."}, {"review_id": "vaEnGmyAgr", "reviewer": "Reviewer_AkhZ", "summary": "This paper introduces an approach known as Successive Layer Training (SLT) for federated learning (FL) on edge devices with limited resources. The core concept of SLT involves training a portion of the neural network (NN) parameters on the devices while keeping the remaining parameters fixed. Subsequently, the training process progressively expands to include larger subsets until all parameters have been trained. The authors assert that SLT offers several advantages, including reduced memory requirements for training, enhanced accuracy and convergence speed in FL, and compatibility with heterogeneous devices that possess varying memory constraints. To validate their claims, the authors conduct evaluations of SLT on multiple datasets and NN architectures. Additionally, they compare SLT against state-of-the-art techniques such as Federated Dropout (FD), FedRolex, HeteroFL, and FjORD.", "questions": "In the evaluation, the used model sizes are relatively large and can be over-parameterized for the dataset. As a result, the method Small Model, which is still sufficient for the task, can achieve better performance than the SOTA methods. If the model size goes smaller (i.e., LeNet with 1x, 2x, or 4x channels), do the proposed method and the method Small Model still outperform the SOTA baselines?", "limitations": "Please see the weakness and questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1, This paper is well-motivated. Memory footprint is a key constraint in training models on edge devices. The proposed approach allows for co-adaptation between parameters and reduces the memory footprint.  \n2, The paper is well-written, offering clarity and organization.  \n3, This paper conducted a comprehensive experimental analysis on four datasets (CIFAR10, CIFAR100, FEMNIST, and TinyImageNet) and three NN architectures (ResNet20, ResNet32, and DenseNet40). Comparisons are made with several baselines, and the results demonstrate that SLT achieves higher accuracy and faster convergence in both independent and non-independent and identically distributed (iid and non-iid) settings. Additionally, the performance of SLT is investigated in heterogeneous environments, where devices have different memory constraints.  \n", "weaknesses": "The paper primarily focuses on addressing memory constraints in federated learning (FL) by introducing the Successive Layer Training (SLT) technique. However, it does not thoroughly consider other types of resource constraints, such as computation or communication, which are also crucial aspects of FL on devices. The paper does not provide insights into how SLT impacts computation time or communication costs in the FL process. It is important to acknowledge that SLT may introduce certain overheads or inefficiencies in these areas, potentially influencing its practical applicability and overall performance. Further investigation and analysis of the computational and communication implications of SLT would provide a more comprehensive understanding of its practicality and effectiveness in resource-constrained scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We like to thank the reviewer for the fair and constructive feedback on our manuscript.\n\n# Model capacity\n## Model capacity in conducted experiments\nThe observation is correct that in some experiments the NN architecture is over-parametrized for the given dataset. We observe this, especially for the FEMNIST dataset from the Leaf benchmark. Here, using a memory constraint of $0.5\\times$, only reduces the accuracy from $87.6\\\\%$ to $86.9\\\\%$. However, we observe that if the dataset is more complex and capacity is not sufficient, SLT shows larger gains over a small model (e.g., when using tinyimagenet and ResNet32).\n## Effectiveness of SLT and baselines with small models\nIn general, we observe that SLT can reduce the memory requirement (relative to the full NN) more effectively if the NNs have more layers, as the minimum memory SLT requires for training is mainly dominated by the memory cost of *filling up* a single layer. Hence, when having only a few layers, as it is the case with LeNet, SLT is not as effective w.r.t. memory reduction, and hence we do not expect SLT to perform much better than a small model. Additionally, we confirmed that the trends of the gap between a small model and FedRolex still exist, even when capacity is sufficiently lower. We Evaluated a minimal ResNet structure with only 9 layers and scaled-down memory using $0.125\\times$. We observe that the small model reaches an accuracy of $57.7\\\\%$, while FedRolex only reaches $25.6\\\\%$.\n\n# Communication and computation in SLT\nWe differentiate between memory constraints and constraints on communication and computation (FLOPs), as the latter are less restrictive constraints (e.g. referred to as soft constraints in [1]). Specifically, insufficient available memory excludes a device from the training (as it is done with Google GBoard training [2]), while constraints on communication and FLOPs can slow down the convergence (e.g., w.r.t. time or energy). Hence, when dealing with communication and FLOPs resources, one should opt for using them efficiently.\n\n### Communication efficiency of SLT\nWe want to point out that we already consider communication efficiency in our evaluation (Figure 4). We show (Figure 4: x-axis is the uploaded volume of the data to the server, and hence the communication overhead) that we require significantly less communication (GBytes) to reach the same accuracy as state of the art. Also, with respect to the communication overhead, we converge as fast as a small model but enable higher final accuracies. This is in part due to the freezing of layers, as the parameters of frozen layers do not change, and hence do not need to be uploaded to the server, increasing the communication efficiency.\n\n### Computation (FLOPs) Efficiency of SLT\nWe performed a similar evaluation to show the FLOPs overhead in federated training for SLT and the related baselines. We observe similar trends: 1) we require significantly fewer FLOPs to reach a certain level of accuracy, compared with state of the art; and 2) the number of FLOPs required by SLT to reach a certain accuracy is similar to the small model, but SLT enables higher final accuracies. The results are shown in **Figure R.2** in the rebuttal pdf.\n\nWe would add these results to a revised version of the manuscript.\n\n[1] K. Pfeiffer, M. Rapp, R. Khalili, J. Henkel, \"Federated Learning for Computationally-Constrained Heterogeneous Devices: A Survey\", ACM Computing Surveys, Volume 55, Issue 14, 2023  \n[2] Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel\nRamage, and Françoise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv:1812.02903, 2018."}, {"review_id": "CdhlcUhP9k", "reviewer": "Reviewer_tzDU", "summary": "The paper proposes an FL training methodology to reduce the memory footprint for constrained devices. Specifically, the layers inside the original model are divided into three categories, i.e., frozen, fully trained, and partially trained. The frozen layers only participate in the forward pass but do not require storing the activations or computing gradients for the backward pass. The partially trained layers also only train a fraction of the parameters, thus reducing the memory overhead. The authors further provide heuristics for determining the configuration of layers within these three categories throughout training and the number of training steps allocated to each configuration.", "questions": "Please see weaknesses above.", "limitations": "The authors do not discuss the potential negative societal impact of the work in the main text.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written, easy to follow, and the ideas are clearly presented. \n- The experiments investigate the method from different aspects like iid versus non-iid data distribution as well as iid versus non-iid compute devices, and show benefits over several baselines.", "weaknesses": "- While the proposed method works well on the evaluated benchmarks, it appears to be a straight-forward combination of existing ideas (using s ratio of parameters to partially train layers + progressive freezing). As such, the reviewer finds the originality of the ideas rather limited.\n- Can the authors provide some intuition or theoretical justification for acceptable bounds on s and the general configuration of F_F, F_T, and F_H? Specifically, what is the bound on these design hyperparameters after which the convergence is impeded?\n- There seems to be a large gap between the performance of prior work (FedRolex) versus the performance presented in the original paper. Can the authors clarify any difference in the experimental setup that may have led to this?\n- It would be beneficial to present the corresponding training memory footprint (activation+parameters+grads) for each benchmark to better clarify what constraints each benchmarked s_fd ratio corresponds to, and see e.g., how aggressive is 0.125 ratio.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We like to thank the reviewer for the fair and constructive feedback on our manuscript.\n\n# SLT design choices ($K_T, K_F, s$)\nOur general choice of setting $K_T = K_F + 1$ while maximizing $s$ throughout all configurations $N$ is motivated by preliminary ablation studies we conducted (please refer to **Figure R.3** in the rebuttal pdf). Specifically, we investigated, given a specific memory budget, should $K_T$ or $s$ be maximized, as increasing both increases the memory consumption in training. We evaluated the accuracy in a centralized training setup (no freezing applied, i.e. $K_F=0$). The results suggest that maximizing $s$ in favor of $K_T$ clearly results in higher accuracies, therefore, we only *fill up* and train a single layer per step (i.e. the minimal possible number of layers). $F_H$ is given by the design choice of $K_T$.\n\n# SLT Memory limit / minimal $s$\nThe memory reduction limit of SLT is in general affected by two things:\n* Firstly, the limit is constrained by how much memory it requires to *fill up* and train a single layer of the NN (relative to full training). Note that the memory cost of training a single layer compared to the training of the full NN decreases as we have more layers.\n* Secondly, it is influenced by how low $s$ can be set. Note that the subset of a layer of the head $w$ is defined by $s$, ($w \\in \\mathbb R^{\\lfloor sP \\rfloor \\times \\lfloor sM \\rfloor}$). Hence, the minimum of $s$ would be determined by $P$ and $M$, since at least a single filter per layer has to be trained. Consequently, if a layer only has 6 output filters, the minimum applicable $s = \\frac{1}{6}$.\nSLT allows for memory reduction of $0.125\\times$ for ResNet20, $0.09\\times$ for ResNet32, and $0.08\\times$ for ResNet56.\n\n# Accuracy difference between our implementation of FedRolex and the FedRolex paper\nWe are fairly confident that our implementation of FedRolex reflects what is described in the original paper since we used the paper and the authors' public source code for implementation. We additionally ran our implementation against theirs and confirmed that throughout the rounds the same subset of parameters get modified on the FL device. We think the differences in the FedRolex paper (Figure 4) are caused by the following aspects: \n\n* In FedRolex, the authors use a significantly larger NN model to evaluate CIFAR10 (generic ResNet18, $11$M parameters) while we use the specifically adapted version for CIFAR10 (ResNet20, $0.27$M parameters).\n\n* FedRolex scales down the NN with $\\gamma=\\{2,4,8,16\\}$, where $\\gamma$ refers to **size** (parameters) reduction of the global model, while we evaluate the reduction of memory that is mainly dominated by the activations. As a consequence, when scaling down the NN with $s$, the size of the NN reduces quadratically, while memory only reduces linearly.\n\nTo demonstrate this, we added $\\gamma$ and memory scaling into a table, comparing the accuracy results from Figure 3 of the FedRolex paper and our results from Figure 1.\n| $\\gamma$ | $1$ | $2$ | $4$ | $8$ | $16$ |\n|---|---|---|---|---|---|\n| memory reduction | $1$ | $1.41$ | $2$ | $2.82$ | $4$ |\n| FedRolex (paper, Figure4) | $84.5\\\\%$ | $68\\\\%$ | $58\\\\%$ | $58\\\\%$ | $58\\\\%$ |\n| FedRolex (ours, Figure1) | $87.6\\\\%$ | - | $80\\\\%$ | - | $61\\\\%$ |\n\nIn our experiment (Figure 1), we evaluate with IID data while FedRolex evaluates \"low heterogeneity\" of data. The remaining differences are the number of rounds (2000 vs. 2500), learning rate, and learning rate decay schedules. We use sinus-decay, while FedRolex uses fixed steps to lower the learning rate. When comparing full training (i.e. $1\\times$) this already creates an accuracy improvement of ~$3$p.p. for FedRolex. So in practice, our setup allows for (slightly) higher final accuracies.\n\nApart from these, the setup is fairly similar using 100 devices with 10 active per round. We, therefore, think the differences in accuracy mostly come down to the choice of the NN model and the distribution of data.\n\n# Memory footprint of models and experiments\nGenerally, the maximum amount of memory for training is dominated by the activations. With scaling down the number of filters using $s$, the activation maps linearly reduce. The number of parameters and respective gradients however reduce quadratically. The following tables show the maximum memory consumption during training and the respective contributions of activations, weights, and gradients for end-to-end training of the NN and scaled-down models (small model).\n\n|**Full NN training** |ResNet20|ResNet32|DenseNet40|\n|---|---|---|---|\n|activations (MB)|$96.0$| $853.8$| $408.7$|\n|weights (MB)|$1.2$|$2.8$|$0.7$|\n|gradients (MB)|$1.2$|$2.8$|$0.7$|\n\n|**Reduction factor $s=0.125$** |ResNet20|DenseNet32|DenseNet40|\n|---|---|---|---|\nactivations (MB)| $12.0$ | $106.7$|   $50.3$\nweights (MB)| $0.02$| $0.05$ |  $0.01$\ngradients (MB)| $0.02$ | $0.05$ |  $0.01$|\n\n\n"}, {"review_id": "QfyRekJwmw", "reviewer": "Reviewer_akhJ", "summary": "The paper focuses on enabling federated learning (FL) among clients that are bounded by the memory capacity. To this end, the work proposes successive layer training (SLT), a technique that sustains the memory usage below a given memory budget throughout the FL process. SLT partitions the model into three parts: *i)* a first part with frozen weights, *ii)* a intermediate part where all weights are trained, and *iii)* a final part (which can stop before the end of the architecture) whose only a fraction of the weights are trained, controlled by a parameter *s* which selects the number of channels to be trained. At training time, the FL process is broken into *N* stages. At each stage, the layers that are included in each of the three parts are changed. Progressively, the final part is moved towards the last layers of the network, until the whole model has been trained. The proposed scheme is applied on CNNs and compared with a number of baselines that employ nested submodels.", "questions": "1) Please specify the number of configuration steps *N* and the duration of each step for the presented experiments.  \n2) See the two points in Weaknesses.  \n3) As the work focuses only on memory constraints and not computational capabilities, it would be better to modify the title to reflect that, *e.g.* \"Aggregating Capacity in FL through Successive Layer Training for Memory-Constrained Devices\" instead of \"Computationally Constrained\".\n", "limitations": "The paper adequately covers its limitations and broader impact. All the main points are discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) The proposed STL technique is interesting. Its parametrisation with respect to the indices of three-part network partitioning and the number of configuration steps provides a high level of flexibility and can thus be adapted to various settings, *e.g.* different memory constraints and clients with significant heterogeneity in their memory capacity.  \n2) The paper is well written, the proposed method is clearly described and positioned within the existing FL approaches.  \n3) The evaluation includes appropriate baselines and covers adequately different setups.", "weaknesses": "1) Althought STL seems to outperform the evaluated baselines, the potential reasons behind this performance gain is not discussed. For example, in the Heterogeneous Memory Constraints setup (Section 3.4), despite being the most realistic, the paper does not discuss why STL performs better than HeteroFl and FjORD, *i.e.* the two strongest baselines for this setup. The *Heterogeneity results* paragraph describes the results of Table 3 in a textual form, but does not attempt to explain why these are as they are.  \n2) The paper does neither discuss the selection of parameter *N*, which determines the number of distinct configurations to be used during the FL process, nor experimentally evaluates its impact. Similarly, there is no investigation on how these configuration steps are distributed across the global rounds. This is briefly touched upon in the last paragraph of Section 3.2, but this is not sufficient. More thorough investigation is needed, for STL to be useful. The key question is how to train using STL, *i.e.* how many configuration steps to use and how much time to spent using each configuration.   ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We like to thank the reviewer for the fair and constructive feedback on our manuscript. We will consider changing the papers title.\n\n# Performance of HeteroFL and FjORD\nWe think the performance of HeteroFL and FjORD (Section 3.4) is limited due to the same effect that is present in FedRolex and FD: the limited co-adaptation that is possible within a layer, and calculating gradients of a subset of the layer without considering the remaining layer weights that reside on the server model. We show in Appendix A that the more distinct subsets are used the lower the accuracy gets, and that even with 4-5 subsets, the accuracy in CIFAR10 can reduce from $80\\%$ to $60\\%$. \n\nHeteroFL and FjORD use in principle the same subset mechanism as FedRolex and FD. Specifically, to lower the resources, constrained devices train a fixed smaller subset that is merged on the server with the remaining parameters. For example in HeteroFL increasing the number of heterogeneity levels from three ($[0.125, 0.25, 0.5]$) to four ($[0.125, 0.25, 0.5, 1.0]$) reduces the accuracy from $24.1$p.p. to $23.3$p.p., despite $\\frac{1}{4}$ of devices training the full NN in the latter case. In SLT however, we avoid that devices train different subsets of a layer. We ensure that each device (independent from resources) trains the same head $F_H$, while allowing stronger devices to freeze fewer layers.\n\n# Number of steps $N$ for different NN models and memory constraints\nThe number of steps $N$ depends on the choice of $K_F$ and $K_T$. For the full argument of why we select $K_T = K_F + 1$, please refer to the answer to reviewer tzDU. As a consequence of this design choice, only a single layer gets *filled up* per step. Therefore, in general, we require as many steps as there are layers in an NN. However, if the memory limit allows within a step to apply $s=1$, all remaining layers can be trained in full width, thus no further step is needed. Consequently, the number of steps also depends on the given memory constraint.\n\nThe following table gives the number of steps for each NN architecture and evaluated constraints.\n| constraint $s$ | ResNet20 | ResNet32 | DenseNet40 |\n|---|---|---|---|\n| 0.66 | - | - | 10 |\n| 0.5 | 8 | 14 | - |\n| 0.33 | - | - | 15 |\n| 0.25 | 12 | 26 | - |\n| 0.125 | 16 | 14 | - |\n\n# Distribution of $N$ steps to $R$ rounds\nThe following describes our methodology to distribute $N$ steps to $R$ rounds. The mapping of $N$ to $R$ for four of our experiments is visualized in **Figure R.1** in the rebuttal pdf. Generally, a layer that is trained should receive sufficient amount of training, to extract useful features for the downstream layers, but at the same time, it should not overfit in the current configuration. Hence, there is a tradeoff between too little training and too much training within a step. \nWe distribute all steps $N$ over total rounds $R$ by calculating the share of parameters that are added to the training when transitioning from step $n$ to $n+1$. We calculate the number of trained parameters in the NN, based on $K_F$, $K_T$, and $s$ using $Q$, s.t.  \n$Q(K_F, K_T, s) = \\bigg( \\sum_{k \\in \\\\{k:K_F < k \\leq K_T\\\\}}  P_k M_k \\bigg) + P_{K_T+1}\\lfloor sM_{K_T+1}\\rfloor + \\sum_{k \\in \\\\{k:K_T+1 < k \\leq K\\\\}}\\lfloor sP_k\\rfloor \\lfloor sM_k \\rfloor$  \nUsing $Q$ we calculate the number of rounds per step $R_N$,       \n$R_n = R \\frac{Q(K_F^{(n)},K_T^{(n)},s^{(n)})}{Q(0,0,1)}$ in case $n = 0$,  \n$R_n = R \\frac{Q(K_F^{(n)},K_T^{(n)},s^{(n)}) - Q(K_F^{(n-1)},K_T^{(n-1)},s^{(n-1)})}{Q(0,0,1)}$ else  \nand lastly, the switching points $r_n$ by using  \n$r_n = R_n + \\sum_{i=0}^{n-1} R_j$.\n\nWe observe that distributing the rounds based on the number of added parameters produces robust results throughout all experiments with fast convergence and high accuracy. We have also evaluated other empirical approaches like sharing rounds equally among rounds, which resulted in slower convergence and did result in lower final accuracy (e.g., $-3\\text{p.p.}$ for CIFAR10). Additionally, we considered dynamic approaches, that switch based on the test accuracy/loss on the server (a similar technique has been used by Kundu and Jaja [1]). However, such approaches require dataset-specific hyperparameter tuning, and our results show that they do not produce necessary better results, as shown in **Figure R.4** in the rebuttal pdf.\n\n\n  \n[1] Amit Kumar Kundu and Joseph Jaja. \"FedNet2Net: Saving Communication and Computations in Federated Learning with Model Growing.\" International Conference on Artificial Neural Networks. Cham: Springer Nature Switzerland, 2022."}]

Author rebuttal:
We attached a pdf with four figures (R.1, R.2, R.3, R.4) to support our answers to the reviewers' questions.

Meta-review: This paper considered the memory restricted scenario while training a large DL model. It proposed a block coordinate descent alike algorithm to solve such challenge. This paper has included extensive experiments to justify the proposed approach. All reviewers are positive to accept this paper. The following aspects can be polished in the final version:

- provide explanation to show the convergence and correctness of the proposed approach.
- add the makeup experiments in this paper
- also discuss the computation and communication restricted scenarios.
Decision: Accept (poster)