SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Equal distribution: In this scheme, we equally distribute the rounds to the steps, i.e.,  $R_{n} = \frac{R}{N + 1}$ .
- Early stopping: In this scheme, we decide online, based on the test accuracy, when to switch. If the test accuracy on the server for a number of FL rounds does not improve, the mapping switches to the next configuration. The number of rounds is usually referred to as patience. We evaluate with patience 5, 15, and 25.
To compare the mapping schemes, we run experiments with CIFAR10 and ResNet20, where, except for the mapping scheme, all hyperparameters are kept the same (as presented in Section 3). We can observe from the results in Fig. 8 that our mapping scheme outperforms the others with respect to the final accuracy and convergence. Even though the early-stopping-based technique with patience 5 increases  $n$  more aggressively, it does not result in faster convergence or higher final accuracies.
D Miscellaneous Experiments
To evaluate how SLT and baselines perform with more complex datasets and deeper NNs, we evaluate it with the full ImageNet [43] dataset (1.28M images, 1K classes). However, we downscale the
Figure 7: Accuracy over rounds (green) and steps over rounds (black) in SLT for CIFAR100/DenseNet40, CIFAR10/ResNet20, and TinyImageNet with ResNet44.
SLT
SLT (equal share)
SLT (early stopping 5)
SLT (early stopping 15)
SLT (early stopping 25)
Figure 8: Different strategies for mapping  $N$  to  $R$  using CIFAR10 with ResNet20 and  $s_{\mathrm{FD/FedRolex}} = 0.25$ . Accuracy over rounds is displayed on the left. Steps over rounds are displayed on the right.
images to  $3 \times 64 \times 64$  pixels to reduce the complexity of the evaluation. To account for the larger dataset, we increase the number of devices to  $|\mathcal{C}| = 500$  and the number of rounds to  $R = 4000$ . All remaining hyperparameters are kept the same (Section 3). We use ResNet56 to account for the more complex dataset. Results for iid and non-iid data are provided in Table 5. We observe that the general trend of Tables 1 and 2 remains the same: SLT outperforms the state of the art and the small model baseline with large margins.
E Training Memory Measurements in PyTorch
We measure the maximum memory requirement for the evaluated NN models ResNet and DenseNet using PyTorch 1.10. Specifically, we measure the size of the activations, gradients, and weights. These memory measurements are done offline (prior to training) and do not require any data.
- Measurement of weights: To measure the size of the weights, we sum up all tensors that are present in the NN's state_dict.
Table 5: Results for iid and non-iid experiments with ResNet56 usingImagenet  $(64\times 64)$  are given. Accuracy in  $\%$  after 4000 rounds of training is given.
Setting
sFD/FedRolex	ResNet56/ImageNet/iid	ResNet56/ImageNet/non-iid
0.125	0.25	0.5	1.0	0.125	0.25	0.5	1.0
SLT (ours)	24.2±0.2	31.2±0.3	34.6±0.1	21.7±0.9	29.7±0.3	31.8±0.4
Small model	8.9±0.2	18.4±0.0	30.3±0.1	41.6±0.5	8.4±0.3	16.2±0.2	27.3±0.3	38.7±0.3
FedRolex [20]	3.4±0.2	11.4±0.3	21.3±0.5	2.8±1.1	10.2±0.9	18.4±0.4
FD [9]	0.3±0.2	6.2±0.2	16.8±0.5	0.1±0.0	0.1±0.0	15.7±0.4
- Measurement of activations and gradients: To measure the size of the activations that have to be kept in memory, as well as the gradients, we apply backwardHooks to all relevant PyTorch modules in an NN. Specifically, we add these hooks to Conv2d, BatchNorm2d, ReLU, Linear, and Add operations. If a hook attached to a module is called, we add the respective size of the activation map and the size of the calculated gradient to a global variable to add up all required activations and gradients.