SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Results for iid experiments with ResNet and DenseNet using CIFAR10, FEMNIST, CIFAR100, and TinyImageNet. Accuracy in % after  $R$  rounds of training is given.
Setting
sFD/FedRolex	ResNet20/CIFAR10	ResNet20/FEMNIST
0.125	0.25	0.5	1.0	0.125	0.25	0.5	1.0
SLT (ours)	74.1±0.8	83.4±0.2	85.2±0.6	84.4±0.3	85.8±0.1	86.9±0.0
Small model	66.3±0.3	78.2±0.4	84.6±0.4	87.5±0.6	82.3±0.4	85.5±0.1	86.9±0.0	87.6±0.0
FedRolex [20]	21.7±1.9	61.0±0.5	80.6±0.6	42.1±6.0	71.4±2.1	83.0±0.1
FD [9]	19.0±1.4	36.7±1.8	71.6±0.4	38.9±3.7	70.4±0.2	83.4±0.3
Setting
sFD/FedRolex	DenseNet40/CIFAR100	ResNet44/TinyImageNet
0.33	0.66	1.0	0.125	0.25	0.5	1.0
SLT (ours)	51.1±0.4	53.3±0.6	33.5±0.1	40.3±0.5	42.3±0.2
Small model	43.9±1.5	55.9±0.1	60.2±0.5	19.8±0.3	30.6±0.3	40.2±0.3	45.2±0.1
FedRolex [20]	22.2±0.3	46.7±0.1	6.9±0.2	19.8±0.8	33.1±0.2
FD [9]	13.5±0.5	41.9±1.5	0.9±0.0	7.1±0.1	25.9±0.6
Preliminary experiments have shown that this mapping scheme outperforms other techniques, like an equal distribution of rounds to all configurations, and enables SLT to converge as fast as a small model while reaching a significantly higher final accuracy (we provide further results in Appendix C). The mapping of steps  $N$  to rounds  $R$  does not rely on private data (or any data) and can be stored in a look-up table prior to the training. A visualization of SLT is given in Fig. 3. We provide the number of steps  $N$  for different NN architectures and constraints in Appendix C.
3.3 Experimental Results
Iid results: For the iid case, results are given in Table 1. We observe that SLT reaches significantly higher accuracy for ResNet20 and CIFAR10 for all evaluated constraints, outperforming a small model baseline by up to 7.8 p.p. and state of the art by 52.4 p.p.. The results with FEMNIST show that a small model baseline already provides sufficient capacity for the dataset since only a few percentage points separate  $s_{\mathrm{FD/FedRolex}} = 0.125$  from the full model (i.e., when  $s_{\mathrm{FD/FedRolex}} = 1$ ). Hence, SLT can only provide a minor benefit over a small model baseline. The contrary can be observed for CIFAR100 and TinyImageNet, where using a small model ( $s_{\mathrm{FD/FedRolex}} = 0.125$ ) loses up to 25.4 p.p. to the full model. Additionally, it can be observed that for low memory constraints, FD and FedRolex fail to learn a useful representation at all. SLT improves upon a small model by up to 13.7 p.p. and up to 26.6 p.p. compared to state of the art.
Non-iid results: Typically, data in FL is not distributed in an iid fashion but rather non-iid. We repeat the experiments shown in Table 1 but distribute the data on the devices in a non-iid fashion. Similar to [27], we apply a Dirichlet distribution, where the rate of non-iid-ness can be varied using  $\alpha$ . For all experiments, we set  $\alpha = 0.1$ . We observe from Table 2 that the small model baselines in the case of CIFAR10 and FEMNIST lose accuracy compared to the full model. Hence, the gain of SLT compared to a small model baseline increases. The results for CIFAR100 and TinyImageNet show a proportional drop in accuracy for all algorithms. However, SLT still outperforms other techniques by a large margin. Note that we could apply common non-iid mitigation techniques like FedProx [11] on top of SLT to further limit the drop in accuracy.
For additional experimental results, we refer the readers to Appendix D.
Communication, computation, and convergence speed: We evaluate our technique w.r.t. the communication overhead of the distributed training and the number of computations devices have to perform (FLOPs). Specifically, we evaluate the gain in accuracy over required transmitted data and performed FLOPs. We show the results in Fig. 4. We observe that our technique converges fast, similarly to a small model, while reaching higher final accuracy. Compared to FD and FedRolex, our technique requires significantly less communication to reach the same level of accuracy. Similar behavior can be observed w.r.t. FLOPs.
3.4 Heterogeneous Memory Constraints
Memory constraints in devices can be heterogeneous. We evaluate SLT in such scenarios and compare it against the start of the art. We evaluate with different resource levels and split the available constraint
Table 2: Results for non-iid experiments with ResNet and DenseNet using CIFAR10, FEMNIST, CIFAR100, and TinyImageNet. Accuracy in % after  $R$  rounds of training is given.
Setting	ResNet20/CIFAR10	ResNet20/FEMNIST
sFD/FedRolex	0.125	0.25	0.5	1.0	0.125	0.25	0.5	1.0
SLT (ours)	52.4±0.9	69.6±0.6	75.5±1.3	81.2±1.6	83.0±2.0	83.8±1.9
Small model	44.7±1.2	63.1±0.7	73.6±0.6	80.5±1.3	79.6±0.8	82.9±1.1	83.3±2.4	84.0±1.9
FedRolex [20]	15.0±3.7	29.8±1.7	48.3±2.9	39.4±2.0	59.3±2.1	78.5±0.5
FD [9]	11.3±0.9	10.7±0.6	34.9±5.7	15.9±8.2	51.0±1.2	79.7±1.1
Setting	DenseNet40/CIFAR100	ResNet44/TinyImageNet
sFD/FedRolex	0.33	0.66	1.0	0.125	0.25	0.5	1.0
SLT (ours)	45.9±1.4	48.4±0.5	28.5±1.2	35.1±1.1	36.1±0.2
Small model	40.5±1.2	51.8±0.2	55.8±0.5	16.9±0.2	25.3±0.5	34.2±0.4	39.0±0.8
FedRolex [20]	20.0±0.3	42.9±0.4	1.5±0.5	12.7±1.3	26.1±0.5
FD [9]	7.6±0.1	36.9±1.0	0.4±0.1	0.6±0.0	20.0±1.3
Figure 4: Maximum reached accuracy in % over data upload and performed FLOPs for CIFAR10, CIFAR100, and TinyImageNet using DenseNet40, ResNet20, and ResNet44 in an iid training case.
levels equally upon the devices, i.e., when constraints of  $s_{\mathrm{FD/FedRolex}} = [0.125, 0.25]$  are given,  $50\%$  of the devices train with  $s_{\mathrm{FD/FedRolex}} = 0.125$  while the remaining  $50\%$  use  $s_{\mathrm{FD/FedRolex}} = 0.25$ . SLT supports heterogeneous constraints through the following mechanism: Firstly, devices with the highest constraint perform training as done in the homogeneous case, outlined in Algorithm 1 using the head's scale factor as described in Eq. (2). Devices that are less constrained use the same scale factor  $s_n$  per configuration to ensure that all devices train the same number of parameters within a layer. To utilize the remaining available memory, less constrained devices freeze fewer layers, therefore, train more layers at full width. For a given  $K_T$  and  $s_n$  of a configuration  $S_n$ , the remaining memory of less constrained devices is utilized by minimizing  $K_F$ , s.t.
$$
\min  K _ {F}, \text {s . t .} \operatorname {m e m o r y} \left(S _ {n}\right) \leq \operatorname {m e m o r y} \left(s _ {\mathrm {F D / F e d R o l e x}}\right). \tag {7}
$$
In addition to FD and FedRolex, we evaluate HeteroFL [18] and FjORD [19]. Both require that some devices are capable of training the full NN end-to-end, otherwise, some parameters do not receive any updates. In cases where no device can train the server model end-to-end, we reduce the size of the server model such that at least one participating device can fully train the model.
Small model: All devices train a small model regardless of their constraints. Scale factor  $s$  is set to the minimum a participating device supports.
FedRolex [20]: Similar to the homogeneous case, FedRolex uses a rolling window (Eq. (3)). In heterogeneous cases, devices use a constraint-specific scale  $s_e$ , to adjust the number of filters  $\lfloor s_e M_k \rfloor$ .
Table 3: FL with heterogeneous constraints. Accuracy in % after  $R$  rounds of training is given.
Setting	DenseNet40/CIFAR100	ResNet44/TinyImageNet
sFD/FedRolex	[0.33, 0.66]	[0.33, 0.66, 1.0]	[0.125, 0.25]	[0.125, 0.25, 0.5]	[0.125, 0.25, 0.5, 1.0]
SLT (ours)	46.4±2.0	49.3±1.8	30.3±1.2	33.0±0.5	35.9±0.4
Small model	40.5±1.2	40.5±1.2	16.9±0.2	16.9±0.2	16.9±0.2
FedRolex [20]	33.2±0.4	43.9±1.3	5.4±0.2	13.8±1.3	23.6±0.7
FD [9]	21.2±0.6	38.1±0.4	0.5±0.1	0.6±0.1	20.6±1.7
HeteroFL [18]	42.2±1.3	42.8±0.5	20.7±0.7	24.1±0.2	23.3±0.5
FjORD [19]	38.7±0.4	36.9±0.4	22.4±1.0	25.3±0.3	27.5±0.8
FD [9]: Although Caldas et al. [9] do not specifically evaluate heterogeneous devices, heterogeneity can be supported straightforwardly by using constraint-specific  $s_e$  for scaling down the NN. This extension of FD is also evaluated in FedRolex and FjORD.
HeteroFL [18]: In HeteroFL, devices use the same subset throughout the training. To support heterogeneity, devices use a resource-specific scaling factor  $s_e$ , s.t. for each  $s_e$  the indices are selected using  $I_k^{(r,e)} = I_k^{(e)} = \{i \mid 0 \leq i < \lfloor s_e M_k \rfloor\}$ .
FjORD [19]: FjORD uses the same indices as HeteroFL, but each device switches between constraint-specific subsets that satisfy the device constraints within a local epoch on a mini-batch level.
Heterogeneity results: We repeat the experimental setup as presented in Table 2 for TinyImageNet and CIFAR100, but enforce varying device constraints in the experiments (see Table 3). We observe that SLT outperforms others in all evaluated scenarios. FedRolex can improve upon a small model in some settings, but this is not the case with FD. For FjORD and HeteroFL, we observe that both outperform the small model baseline. Yet, in some cases, both HeteroFL and FjORD have a lower accuracy when utilizing more constraint levels. For HeteroFL, it can be observed that using ResNet44 with 4 constraint levels reaches a lower accuracy than with 3 levels (despite the fact that all devices have higher average resources  $\mathbb{E}[s_{\mathrm{FD / FedRolex}}]$  of  $\approx 0.47$  in the case of [0.125, 0.25, 0.5, 1.0] instead of  $\approx 0.29$  in the case of [0.125, 0.25, 0.5]). The same can be observed for FjORD with DenseNet40. As both techniques, in principle, use the same subset mechanism as FedRolex and FD, we think that both suffer from supporting more constraint levels that cause less co-adaptation between NN filters.
We cover related work that studies similar problems or employs similar techniques.
Resource constraints in FL: Most works on resource-constrained FL target communication. Specifically, the use of quantization and compression in communication [9, 8] and sketched updates [10] have been proposed to lower the communication burden. Chen et al. [28] propose adaptive parameter freezing as they discover that parameters stabilize during training and do not have to be transferred to the server. Another branch of work focuses on reducing communication, computation, and memory requirements by employing only a subset of the full NN on devices. Caldas et al. [9] introduce  $FD$ , a mechanism that creates a device-specific subset by randomly selecting a subset of CNN filters. Diao et al. [18] introduce HeteroFL, which allows for heterogeneous constraints by employing fixed subsets of different sizes to the NN and aggregating them on the server. Horvath et al. [19] introduce a similar technique (FjORD), with the main difference that in FjORD, each device trains every available subset within its capabilities. Rapp et al. [29] propose DISTREAL, a technique that uses varying subsets on a mini-batch level such that devices finish their update on time despite having intra-round changing resources. FedRolex [20] supports heterogeneity similar to FjORD and HeteroFL but allows for server NN models that are outside of the capabilities of all devices. This is enabled by not training a fixed subset of the NN parameters but by training a rolling window of all parameters that is shifted on a round basis. Beyond subsets, the use of low-rank factorization [30, 31] has been proposed to train NN models. Lastly, Qui et al. [32] propose sparse convolutions to lower the resource requirements for training but require special hardware for sparse computations to realize the gains.
Layer-wise model training: Layer-wise model training has been proposed in centralized training of CNNs as an alternative to training with end-to-end backpropagation of the error. Hettinger et al. [33] introduced a technique that adds CNN layers one at a time during training using auxiliary
heads for classification while freezing early layers. Similar techniques have also been employed for unsupervised learning, where representations are trained with contrastive techniques without requiring end-to-end gradient propagation [34, 35]. Recently, the concept of progressive model growth has also been proposed for FL: Wang et al. [36] propose ProgFed, where they discover that by progressively adding CNN layers to the NN while using an auxiliary head, the FL training converges faster and required less communication to reach the same accuracy as an end-to-end baseline. Similarly, Kundu et al. [37] propose a technique that grows the model depending on the complexity of the data to reach a high accuracy if the NN capacity is not sufficient for the problem. Importantly, both techniques only focus on increasing the convergence speed. Hence, they consider communication and computation overhead but not the problem of constrained memory on edge devices, nor do they support heterogeneous devices. In both techniques, eventually, all devices have to train the full-size NN and, consequently, have to have the memory resources available for that.
Memory-efficient training: Several techniques have been proposed to train an ML model in a memory-efficient way. Kirisame et al. [38] present Dynamic Tensor Rematerialization that allows recomputing activation maps on the fly. Similarly, encoding and compression schemes [39, 40] have been proposed to lower the size of the activation maps during training. Techniques like that trade memory for computation, and some lower the accuracy by using approximation or lossy compression. Importantly, these techniques are orthogonal to SLT, FD, FedRolex, HeteroFL, and FjORD.
We proposed SLT that is able to reduce the memory requirements for training on devices, efficiently aggregating computation capacity and learning from all available data. Through extensive evaluation, we show that gains in final accuracy as well as the faster convergence speed (compared with state of the art) are robust throughout different datasets, data distribution, and NN topologies.
Limitations: We observe that SLT is most effective if the used NN architecture is deep (i.e., has many layers), as the cost of filling up a single layer becomes less significant. Also, SLT is less effective if the size of the activation map is strongly unevenly distributed throughout the layers (DenseNet), as it has to adapt to the layer with the highest memory requirements when filled up. Besides, we applied SLT to CNN topologies only. Finally, we mainly focused on memory as a hard constraint [41] for training. We show that communication and FLOP efficiency are significantly higher than in the state of the art, but we did not consider per-round communication or FLOP constraints. For future work, we want to extend our study to other topologies, such as transformers, and employ neural architecture search (NAS) techniques to find NN configurations that reach the highest accuracy when trained in FL with SLT in heterogeneous environments.
Broader impact: Our solution could help reduce biases in FL systems, improving fairness. For instance, by including users with low-end smartphones in the learning process, it provides these users (who perhaps cannot afford high-end devices) with better experiences as the model is going to be trained over their data, too. It could also reduce the cost of deployment of distributed IoT systems (e.g., sensor networks), as they can be implemented with low-cost devices (or a mixture of low and high-cost devices), enabling, e.g., deployment of larger and more fine-grained monitoring systems. On the negative side, distributing learning over low-end devices that are not particularly designed for training tasks can increase the overall energy consumption of the system. This is an important issue that should be studied in more detail.
This work was partially funded by the "Helmholtz Pilot Program for Core Informatics (kikit)" at Karlsruhe Institute of Technology. The authors acknowledge support by the state of Baden-Württemberg through bwHPC.
[1] Yiqiang Chen, Xin Qin, Jindong Wang, Chaohui Yu, and Wen Gao. Fedhealth: A federated transfer learning framework for wearable healthcare. IEEE Intelligent Systems, 35(4):83-93, 2020.
[34] Sindy Löwe, Peter O'Connor, and Bastiaan Veeling. Putting an end to end-to-end: Gradient-isolated learning of representations. Advances in Neural Information Processing Systems, 32:3039-3051, 2019.
[35] Yuwen Xiong, Mengye Ren, and Raquel Urtasun. Loco: Local contrastive representation learning. Advances in neural information processing systems, 33:11142-11153, 2020.
[36] Hui-Po Wang, Sebastian Stich, Yang He, and Mario Fritz. Progfed: effective, communication, and computation efficient federated learning by progressive training. In International Conference on Machine Learning, pages 23034-23054. PMLR, 2022.
[37] Amit Kumar Kundu and Joseph Jaja. Fednet2net: Saving communication and computations in federated learning with model growing. In Artificial Neural Networks and Machine Learning-ICANN 2022: 31st International Conference on Artificial Neural Networks, Bristol, UK, September 6–9, 2022, Proceedings; Part IV, pages 236–247. Springer, 2022.
[38] Marisa Kirisame, Steven Lyubomirsky, Altan Haan, Jennifer Brennan, Mike He, Jared Roesch, Tianqi Chen, and Zachary Tatlock. Dynamic tensor rematerialization. In International Conference on Learning Representations, 2021.
[39] Animesh Jain, Amar Phanishayee, Jason Mars, Lingjia Tang, and Gennady Pekhimenko. Gist: Efficient data encoding for deep neural network training. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), pages 776-789. IEEE, 2018.
[40] Georgios Georgiadis. Accelerating convolutional neural networks via activation map compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[41] Kilian Pfeiffer, Martin Rapp, Ramin Khalili, and Jörg Henkel. Federated learning for computationally constrained heterogeneous devices: A survey. ACM Comput. Surv., 55(14s), jul 2023.
[42] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958, 2014.
[43] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
A Co-Adaptation in Subset-based FL
Federated Dropout is originally inspired by regular dropout [42], a regularization technique that constrains the capacity of a large NN model by randomly dropping parameters from the training, thereby limiting co-adaptation among parameters of the model. This is essential to improve the accuracy and reduce the over-fitting of parameters, as shown in various studies. FD and FedRolex adopt the dropout technique, removing CNN's filters in a round-based manner. These techniques, however, exercise dropout to its extreme, dropping a large part of filters so that not enough co-adaptation between filters remains. In particular, the gradients for the subset of parameters trained on a device are calculated without consideration of the error of the remaining parameters that reside on the server. These subsets are randomly changing over time and devices, reducing the co-adaptation of this distributed training process. Add to these the fact that the data is also distributed over devices, so applying such a random scheme significantly decreases the chance that a subset of parameters is being trained together over a sizable proportion of the data.
To further study the effects of co-adaptation on the reachable accuracy and the differences between FD and FedRolex, we run the following experiment, using CIFAR10 with ResNet20 and  $s_{\mathrm{FD/FedRolex}} = 0.25$ :
- We modify FD s.t. all devices train the same random subset per round, i.e., the same indices  $I^{(r,c)} = I^{(r)}$  per round (index  $k$  is omitted for simplicity).
- We limit the randomness, where at each round, we arbitrarily select  $I^{(r)}$  out of a set  $\mathcal{I} = \{I_1, \ldots, I_{|\mathcal{I}|}\}$  of randomly initialized subsets that are generated once prior to training.
Small model
FedRolex [20]
FD [9] with limited randomness
|I|
Figure 5: FD with limited randomness using CIFAR10 with ResNet20 and  $s_{\mathrm{FD/FedRolex}} = 0.25$ .
We vary the randomness by varying the number of sampled subsets in  $\mathcal{I}$ , s.t.  $|\mathcal{I}| \in [0, 500]$ . Thereby, the probability of a specific subset being selected is  $p = \frac{1}{|\mathcal{I}|}$ . Recall that all the devices train the same submodel in a round, and there are only  $|\mathcal{I}|$  submodels that would be trained by devices over the training period. Evaluation is done with the full server model. Results are shown in Fig. 5.
We observe the following effects: 1) The final accuracy drops proportionally to  $\sim p$ . 2) In the case of  $|\mathcal{I}| = 1$ , FD behaves similarly to the small model baseline, as always the same subset is used for training. We also observe that remaining untrained filters have a minor effect on the accuracy when compared with a small model. However, because of these untrained parameters, the model fails to reach higher accuracies as with SLT (see Section 3). 3) The accuracy drops with introducing more randomness to the training process (i.e., increasing  $|\mathcal{I}|$ ). This is as co-adaptation among parameters of the model reduces as we increase the randomness. 4) The rolling window approach of FedRolex is a special case of FD with limited randomness (i.e.,  $|\mathcal{I}| = 5$  in this experiment).
B Ablation study maximizing  $s$  over  $F_{T}$
To justify our design choice in Section 2 to maximize  $s_n$  for all steps  $n$ , we conduct an ablation study, where we study the best trade-off between  $s$  and  $F_T$ . In particular, instead of maximizing  $s$ , we only use fractions of the maximized  $s_n$  labeled  $s_{\text{ablation}}$ . We evaluate different values for  $s_{\text{ablation}}$ , i.e.  $\frac{s_{\text{ablation}}}{s_n} \in (0,1]$ . When only a fraction of the maximized  $s_n$  is used in a step, the remaining memory can be used to increase the size of  $F_T$ . We conduct with CIFAR10/ResNet20 and TinyImageNet/ResNet44, where all the hyper-parameters are kept the same as in Section 3.1 (except the changes in  $s_n$  and  $F_T$ ). The final accuracy of SLT is displayed in Fig. 6. For each run, we depict the
Figure 6: Trade-off between maximizing  $s$  over  $F_{T}$ . The results show that maximizing  $s\left(\frac{s_{\text{ablation}}}{s_{n}} = 1\right)$  gives the highest accuracy.
Table 4: Steps  $N$  in SLT for different constraints  $s$  and NN models.
Constraint s	ResNet20	ResNet44	DenseNet40
0.66	-	-	10
0.5	8	14	-
0.33	-	-	15
0.25	14	26	-
0.125	16	14	-
average accuracy and standard deviation for three seeds. The results show that by maximizing  $s$  in favor of  $F_{T}$ , SLT reaches a higher final accuracy.
C Mapping of steps  $N$  to rounds  $R$
Generally, any layer that is trained in SLT should receive a sufficient amount of training to extract useful features for downstream layers, but at the same time, it should not overfit in the current configuration. The mapping of rounds  $R$  to steps  $N$  in SLT is done proportionally to the amount of added (previously untrained) parameters to the training. Since  $N$  depends on how many steps are required until  $s = 1$ , the number of steps depends on the constraint level. In Table 4, we list  $N$  for all experiments and constraints. Fig. 7 visualizes this mapping for DenseNet40, ResNet20, and ResNet44, where SLT's accuracy over rounds is displayed in green while steps over rounds are displayed in black. This mapping scheme has key advantages over other techniques. Most importantly, it depends only on the NN structure and not on the data available on the devices. Hence, it can be calculated offline prior to the training.
We compare our mapping scheme with two other mapping schemes, one offline and one online: