SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 FLuID executing on Centralized Server
1: Input:  $M(w)$  #Model.  $M(w_{t})$  is modelatstept  
2: Output:  $S(w)$  # Sub-models for stragglers  
3:  $N \rightarrow$  All Clients,  $T \rightarrow \emptyset$  #  $T$  stragglers,  $N$  non-stragglers  
4:  $IN \rightarrow \emptyset$  # invariant neurons  
5: while not done do  
6: if  $T$  is  $\emptyset$  then {# Straggler and dropout threshold initialization}  
7: Broadcast  $M(w_{0})$  to each  $C$   
8: Receive  $M(w_{1})$  from every  $C$   
9:  $th = \min(\frac{w_{ijt} - w_{ij(t-1)}}{w_{ij(t-1)})})$  # threshold for Invariant dropout  
10: else {# Straggler and dropout threshold recalibration}  
11:  $S(w_{t}) =$  sub_model_generation(IN,  $M(w_{t})$ )  
12: Broadcast  $S(w_{ti})$  all  $i \in T$   
13: Broadcast  $M(w_{t})$  all  $i \land \notin T$   
14: Receive  $\Delta(w_{t+1})$  from every  $C$   
15:  $M(w_{t+1}) = \text{aggregate}(M(w_{t}), \Delta(w_{t+1}))$   
16:  $IN =$  identify_invariant_neurons(th,  $\Delta(w_{t+1})$ ,  $N$ )  
17: end if  
18:  $T, N, L =$  determine_stragglers(  $C$ ) #  $L =$  train latencies  
19:  $T_{target} =$  identify_nextslowest_client(  $L$ )  
20:  $Speedup_i = \frac{T_{straggler_i}}{T_{target}}$   
21:  $r_i, S_i =$  calculate_submodel_size(Speedup_i) #  $r_i$  dropout rate  
22:  $th = \text{increment_threshold}(r_i)$   
23: done = check_training_done(  $M(w_{t})$ )  
24: end while
In order to identify neuron drop candidates, the global server cannot rely on updates from all clients since stragglers only train on and update the sub-model. Instead, the server takes advantage of the fact that non-stragglers train on the complete model and can identify neurons whose weight updates fall within the threshold  $(th)$  for each calibration step. FLuID prioritizes dropping neurons on stragglers whose weight updates fall within the threshold  $(th)$  for the majority of non-stragglers.
The initial threshold value  $(th)$  in the FLuID framework is set as the average of the minimum percent update of all neurons in the initial few training epochs. The threshold is incrementally increased after each epoch until the number of neurons below the threshold is greater than or equal to the number of neurons to be left out of the sub-model. FLuID can have a different drop threshold for each layer. The algorithm targets neurons for dropping whose gradients consistently fall below the threshold over multiple epochs, prioritizing the elimination of non-critical neurons.
This entire process is repeated to recalibrate the stragglers, the drop rate  $(r)$ , and the threshold  $(th)$ .
6 Evaluation Setup
Models and datasets. We evaluate FLuID on three models and datasets as used by the prior works in the federated learning space [MGZP22, JWK+22, HLA+21, CKMT18].
The FEMNIST datasets consist of images of numbers and letters, partitioned based on the writer of the character in non-IID setting. We train a CNN with two 5x5 CONV layers with 16 and 64 channels respectively, each of them followed with  $2 \times 2$  max-pooling. The model also includes a fully connected dense layer with 120 units and a softmax output layer. The model is trained using a batch size of 10 and a learning rate of 0.004.
The Shakespeare dataset partitions data based on roles in Shakespeare's plays in non-IID setting. We train a two-layer LSTM classifier containing 128 hidden units. We train the model with a batch size of 128 and a learning rate of 0.001.
The CIFAR10 dataset consists of images, partitioned using the same strategy as FjORD  $\left[\mathrm{HLA}^{+}21\right]$  and the IID partition provided by the Flower  $\left[\mathrm{BTM}^{+}20\right]$ . For the real-world mobile devices, we train on the VGG-9 model due to its ability to fit within the resource constraints of all the mobile phones we tested. VGG-9 [SZ15] model architecture consists 6 3x3 CONV layers (the first 2 have 32 channels,
Table 1: Software-Hardware specifications of clients
Device	Year	Android Version	CPU (Cores)
LG Velvet 5G	2020	10	1×2.4 GHz Kryo 475 Prime + 1×2.2 GHz Kryo 475 Gold + 6×1.8 GHz Kryo 475 Silver
Google Pixel 3	2018	9	4×2.5 GHz Kryo 385 Gold + 4×1.6 GHz Kryo 385 Silver
Samsung Galaxy S9	2018	10	4×2.8 GHz Kryo 385 Gold + 4×1.7 GHz Kryo 385 Silver
Samsung Galaxy S10	2019	11	2×2.73 GHz Mongoose M4 + 2×2.31 GHz Cortex-A75 + 4×1.95 GHz Cortex-A55
Google Pixel 4	2019	12	1×2.84 GHz Kryo 485 + 3×2.42 GHz Kryo 485 + 4×1.78 GHz Kryo 485
Table 2: Accuracy comparison of Random Dropout, Ordered Dropout, and Invariant Dropout. The text in **bold** indicates instances when Invariant Dropout showcases the highest accuracy. ( $\mu = \text{mean}$ ,  $\sigma = \text{standard deviation}$ , and  $r = \text{sub-model}$  as a fraction of global model).
Dataset	Dropout Method	r=0.95	r=0.85	r=0.75	r=0.65	r=0.5
Accuracy (μ)	σ	Accuracy (μ)	σ	Accuracy (μ)	σ	Accuracy (μ)	σ	Accuracy (μ)	σ
Shakespeare	Random	43.3	0.1	42.5	0.1	42.4	0.1	41.8	0.2	41.3	0.1
Ordered	42.9	0.1	42.3	0.1	42.2	0.2	41.9	0.1	41.4	0.1
Invariant	43.6	0.1	42.5	0.1	42.6	0.2	42.2	0.2	41.7	0.1
CIFAR10(VGG-9)	Random	57.5	0.1	56.8	0.2	57.0	0.2	57.2	0.2	57.2	0.3
Ordered	57.7	0.1	57.0	0.2	57.6	0.2	57.3	0.1	57.1	0.1
Invariant	58.2	0.1	58.4	0.3	57.1	0.1	57.5	0.2	57.4	0.2
FEMNIST	Random	80.6	0.1	80.5	0.2	80.3	0.2	79.3	0.5	79.2	0.3
Ordered	80.6	0.2	80.5	0.2	80.4	0.2	80.3	0.1	79.7	0.3
Invariant	81.1	0.3	80.9	0.1	80.8	0.2	80.3	0.4	80.1	0.3
followed by two 64-channel layers, and lastly two 128-channel layers), two FC dense layers with 512 and 256 units and a final softmax output layer. We train the model with a batch size of 20 and a learning rate of 0.01. We conduct scalability experiments using the ResNet-18 model, and the results are presented in section 6.1. All the baseline dropout methods and Invariant Dropout are evaluated using the same setup.
System Configuration. Table 1 provides the details of the phones used for the experiments. We evaluate five clients and identify one straggler per training epoch. We connect all our client devices and our server over the same network. All the clients run on Android mobile phones from the years 2018 to 2020.
FLuID is implemented on top of the Flower (v0.18.0)  $\left[\mathrm{BTM}^{+}20\right]$  framework and TensorFlow Lite [Goo] from TensorFlow v2.8.0  $\left[\mathrm{ABC}^{+}16\right]$ . Models are defined using TensorFlow's Sequential API, and then converted into .tflite formats.
Evaluation metrics and baselines. We compare the average training performance (wall-clock time) and accuracy for all workloads and experiments. In each evaluation round, clients receive the global model and report their evaluation accuracy and loss on local data to the server. The server calculates the distributed accuracy and loss by performing a weighted average based on the number of testing examples for each client. We compare FLuID with two established baselines: 1) Random Federated Dropout [CKMT18] and 2) Ordered Dropout from FjORD  $\left[\mathrm{HLA}^{+}21\right]$ .
6.1 Results and Analysis
Accuracy Evaluation We compare accuracy of Invariant Dropout with two baselines, using different sub-model sizes. We trained the models for 100, 250, and 65 epochs for CIFAR10, FEMNIST, and Shakespeare datasets, respectively. Table 2 presents the average achieved accuracy  $(\mu)$  along with the standard deviation  $(\sigma)$  for the three datasets. Specifically, Invariant Dropout outperforms Random Dropout across all three datasets. Compared to Random Dropout, our work achieves a maximum accuracy gain of  $1.6\%$  points and on average  $0.7\%$  point higher accuracy for FEMNIST,  $0.6\%$  point higher accuracy for CIFAR10, and  $0.3\%$  point higher accuracy for Shakespeare datasets.
Invariant Dropout also achieves a higher accuracy against Ordered Dropout across all three datasets, with a maximum increase in accuracy of  $1.4\%$  and an average increase of  $0.3\%$  for FEMNIST,  $0.4\%$
(a) Training time for stragglers before and after FLuID.
Figure 4: Performance evaluation of FLuID
(b) Training time reduction with varying stragglers.
for CIFAR10, and  $0.4\%$  for Shakespeare. The accuracy improvements of Invariant Dropout are statistically significant  $(\alpha < 0.05)$ . Moreover, Invariant Dropout shows less variation in accuracy between runs of the same sub-model size and across all sub-model sizes. This is because invariant Dropout eliminates only invariant neurons, which have little impact on the final model efficiency.
Computational Performance Evaluation Figure 4a shows FLuID's capability to effectively select the sub-model, resulting in a significant reduction in the straggler's training time that almost matches the next slowest client. In the absence of FLuID, the straggler's training time is typically  $10\%$  to  $32\%$  longer than the target time. However, after applying FLuID, the straggler's training time is within  $10\%$  of the target time. Furthermore, it is observed that the overall accuracy of the global model tends to be higher when stragglers train with larger sub-models. Consequently, FLuID selects the largest possible sub-model size that minimizes training time variance across clients. Notably, the performance improvement, unlike accuracy, is influenced by the size of the sub-model and not the specific dropout technique employed. For a more detailed analysis of the impact of sub-model size on training time, please refer to Appendix A.3.
Varying stragglers at runtime. FLuID can recalibrate stragglers. During the experiments in Table 2, we observe that the performance of our mobile clients remained relatively stable, without significant changes in the straggler. However, to evaluate the impact of varying conditions at runtime, we randomly executed certain clients at different points in the training process (25%, 50%, and 75% marks). This was achieved by enabling a client to run the training program as a background process between the specified periods. We observed that FLuID successfully adapted to the variation in stragglers during runtime. Figure 4b demonstrates the overall training time for this experiment. On average, the FLuID framework achieved 18% to 26% faster training time compared to the baseline, and 14% to 18% faster training time compared to selecting a static straggler throughout the entire training process. All results include the overhead of FLuID, which, importantly, is not significant. We observe, FLuID calibration process takes significantly less time (less than 5%) compared to the actual training time. This is because the additional computations required for threshold and sub-model calibration are performed centrally on the server, rather than being distributed to edge devices.
Scalability study. To assess the scalability of FLuID, we conducted experiments using simulated clients ranging from 50 to 100. Each machine runs 10 to 20 clients in parallel. Among all clients, we identified the slowest  $20\%$  as stragglers. We further extend the experiment to run CIFAR10 with ResNet-18 as these are emulated clients on a server and can support relatively larger models than mobile devices. Figure 5 shows the accuracy performance across all three datasets. Overall, Invariant Dropout consistently outperforms Ordered and Random Dropout and maintains a better accuracy profile similar to Table 2. In addition, our dropout technique performs significantly better than completely excluding stragglers from the training process.
We further extended the experiment to assess the scalability of FLuID by 1) clustering stragglers into multiple sub-model sizes (Appendix A.4), 2) Exploring the impact of varying ratios of stragglers in the system (Appendix A.5), and 3) a scalability study with 1000 clients where client sampling is employed as we cannot check for stragglers across 1000 devices individually (Appendix A.6). We demonstrate that FLuID can scale to scenarios involving multiple stragglers with varying computational capabilities by tailoring sub-model sizes for each client. Moreover, when compared to state-of-the-art dropout techniques, Invariant Dropout achieves high accuracies, even as the percentage of stragglers in the system scales up.
(a) Shakespeare - LSTM (50 Clients, 10 stragglers)
(b) CIFAR10 - VGG9 (100 Clients, 20 stragglers)
(c) CIFAR10 - ResNet18 (100 Clients, 20 stragglers)
(d) FEMNIST - CNN (100 Clients, 20 stragglers)
Figure 5: The accuracy comparison of Invariant Dropout with Ordered and Random Dropout as we scale to 50-100 clients with  $20\%$  of the slowest clients being stragglers.
Although FLuID is able to mitigate some impact of the stragglers, it does incur minimal overhead to handle stragglers and maintain system performance. Our evaluation takes this into account but this overhead may increase if straggler performance constantly changes.
FLuID currently only uses pre-defined sub-model sizes mapped to straggler performance, which keeps the framework lightweight and avoids high overhead. However, for future works with varied edge devices, fine-grained sub-model determination may further enhance the work.
Due to rapid technological advancements and device variability in handheld devices, system heterogeneity is prevalent in federated learning. Straggler devices, which exhibit low computational performance, act as the bottleneck. In this paper, we address these issues by introducing a novel dropout technique called Invariant Dropout. Invariant Dropout dynamically creates customized sub-models that include only the neurons exhibiting significant changes above a certain threshold. We build a framework, FLuID, which adapts to changes in stragglers as runtime conditions shift. FLuID effectively mitigates the performance overheads caused by stragglers while also achieving a higher accuracy compared to state-of-the-art techniques.
We thank the anonymous reviewers for their insightful comments. This research was supported in part through computational resources provided by Advanced Research Computing at the University of British Columbia (UBC) [soc]. This work was partially supported by Gift from Google and the Natural Sciences and Engineering Research Council of Canada (NSERC) [funding reference number RGPIN-2019-05059]. The work, in part, was supported by Georgia Tech School of Electrical and Computer Engineering and School of Computer Science. The views and conclusions contained herein are those of the authors. They should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of Georgia Tech, Microsoft, and UBC.
[SZ15] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
$\left[\mathrm{UWH}^{+}21\right]$  Rehmat Ullah, Di Wu, Paul Harvey, Peter Kilpatrick, Ivor Spence, and Blesson Varghese. Fedfly: Towards migration in edge-based distributed federated learning, 2021.
[WQR+22] Jianyu Wang, Hang Qi, Ankit Singh Rawat, Sashank Reddi, Sagar Waghmare, Felix X. Yu, and Gauri Joshi. Fedlite: A scalable approach for federated learning on resource-constrained clients, 2022.
$\left[\mathrm{WUH}^{+}21\right]$  Di Wu, Rehmat Ullah, Paul Harvey, Peter Kilpatrick, Ivor Spence, and Blesson Varghese. Fedadapt: Adaptive offloading for IoT devices in federated learning, 2021.
[WWLZ18] Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang. Gradient sparsification for communication-efficient distributed optimization. In NeurIPS, 2018.
$\left[\mathrm{XFD}^{+}21\right]$  Wenyuan Xu, Weiwei Fang, Yi Ding, Meixia Zou, and Naixue N. Xiong. Accelerating federated learning for iot in big data analytics with pruning, quantization and selective updating. IEEE Access, 9:38457-38466, 2021.
[XKG19] Cong Xie, Sanmi Koyejo, and Indranil Gupta. Asynchronous federated optimization, 2019.
[XYXC21] Zirui Xu, Fuxun Yu, Jinjun Xiong, and Xiang Chen. Helios: Heterogeneity-aware federated learning with dynamically balanced collaboration. In 2021 58th ACM/IEEE Design Automation Conference (DAC), pages 997-1002, 2021.
$\left[\mathrm{ZBW}^{+}22\right]$  Zhengyi Zhong, Weidong Bao, Ji Wang, Xiaomin Zhu, and Xiongtao Zhang. Flee: A hierarchical federated learning framework for distributed deep neural network over cloud, edge and end device. ACM Trans. Intell. Syst. Technol., jan 2022. Just Accepted.
A.1 Evolution of Invariant Neurons
Figure 6: The percentage of 'invariant' neurons as the number of training rounds vary
In this section, we provide an example that some neurons in the server are trained quickly and vary only below a threshold in later iterations. Figure 6 shows the percentages of invariant neurons as the number of training epochs increases. Even after only  $30\%$  of the training rounds are completed,  $15\% - 30\%$  of the neurons become invariant across CIFAR10, FEMNIST, and Shakespeare datasets. For this example, we choose thresholds of  $180\%$ ,  $10\%$ , and  $500\%$ , respectively, for these three datasets and compute their invariant neurons. Sending invariant neurons over to the straggler provides no utility; therefore, these neurons can be dropped. Our work FLuID builds upon this insight.
A.2 Choosing Suitable Threshold
Each model has different characteristics in terms of the magnitude of neuron updates. Therefore, choosing different threshold values results in a different number of neurons classified as invariant. We expanded on our initial findings and studied the effect of threshold value on the number of invariant neurons during training. As expected, a higher threshold value leads to a higher percentage of invariant neurons. Table 3 presents the percentage of invariant neurons observed at different threshold values, and the overall training accuracy of the FEMNIST model, using a sub-model size of 0.75 for the stragglers.
Table 3: Threshold vs accuracy results
Threshold value (%)	Percentage of Invariant Neurons (%)	Accuracy (%)
1	3	80.1
3	6	80.3
5	13	80.5
7	18	80.7
8	22	80.7
10	31	80.5
We observe that to obtain the desired accuracy and mitigate performance bottlenecks of stragglers, it is critical to choose the threshold that has the closest number of invariant neurons as the number of neurons to be dropped for the sub-model. The FLuID framework can automatically tune the threshold for the desired model based on the straggler performance, as described in Section 5.
A.3 Impact of Sub-Model Size on Training Time
In this section, we present evidence that there is a linear relationship between client training time and sub-model size. We evaluated the training time of 5 Android-based mobile phones from 2018 to 2020 outlined in Table 1. The training time is expressed in the percentage of the training time for the
Figure 7: Linear relationship between training time and model size across all three datasets
Sub-model Size
full model size ( $r = 1.0$ ). Across CIFAR10, FEMNIST, and Shakespeare, the training time of all five mobile clients decreases linearly as the sub-model size decreases and falls within  $10\%$  of the sub-model size. Using this insight, FLuID selects a sub-model size  $r$  as the available sub-model, the size that's closest to the inverse of Speedup.
A.4 Additional Experiments with Scalability Studies.
In this section, we show that in the case that the network has multiple stragglers, FLuID does not assume that all stragglers have similar capabilities or select a sub-model for all stragglers based on the slowest device. FLuID can select sub-model sizes for each straggler client based on each client's own capabilities. In this experiment, we cluster devices of similar capabilities into four groups of sub-model sizes. Table 4 presents the accuracy when stragglers are assigned to 4 equal-sized clusters (sub-model size 0.65, 0.75, 0.85, 0.95). The overall accuracy generally lies between assigning sub-model sizes of 0.75 and 0.85 for all stragglers. This way, FLuID can achieve a higher training accuracy with a shorter training time, even with stragglers that are initially more than  $35\%$  slower than non-straggler devices.
Table 4: Accuracy comparison of Random Dropout, Ordered Dropout, and Invariant Dropout as we cluster stragglers into different sub-model size groups.
Random	Ordered	Invariant
CIFAR10	71.7	72.3	72.7
FEMNIST	77.5	77.4	78.2
Shakespeare	53.8	53.9	54.1
A.5 Additional Experiments with Varying Straggler Percentages
In this section, we show that FLuID is capable of handling multiple ratios of stragglers. We have run additional experiments to explore the impact of different ratios of stragglers. One common trend we observed across state-of-the-art techniques and FLuID is that accuracy decreases as the ratio of stragglers is increased as more of the devices are now being trained only on the sub-model. Nonetheless, in all the cases, Invariant Dropout offers the highest accuracy because it is aware of the neuron gradient changes and only drops the least changing neurons. The accuracy results of varying the straggler ratios while using 0.75-sized sub-models are summarized in Figure 8.
(a) Shakespeare - LSTM (50 Clients)
Figure 8: Accuracy of varying the straggler ratios from  $10\%$  to  $40\%$  with 0.75 sub-model size
(b) CIFAR10 - VGG9 (100 Clients)
(c) FEMNIST - CNN (100 Clients)
A.6 Scalability of FLuID with Client Sampling
As the federated learning system scales, FL servers sample a subset of clients participating in each training round. At any point in training, FLuID is capable of recalibrating stragglers and supports dynamic changes during runtime. The ability to adjust to a different set of clients and identify stragglers in every training round enables FLuID to easily incorporate client sampling into its process. We scaled FLuID to 1,000 clients with the FEMNIST dataset for 500 global training rounds. We run with a client sampling ratio of  $10\%$ , as used by the prior works in federated learning spaces such as FjORD  $\left[\mathrm{HLA}^{+}21\right]$ . We present the accuracy results in Table 5 against each sub-model size for Invariant Dropout and the baseline techniques. Invariant Dropout maintains a better accuracy profile than the baselines even when scaled up to 1000 clients while incorporating client sampling.
Table 5: Accuracy comparison of Random Dropout, Ordered Dropout, and Invariant Dropout as for FEMNIST with 1000 clients and client sampling of  ${10}\%$  .
r=0.95	r=0.85	r=0.75	r=0.65	r=0.40
Random	87.9	87.5	87.5	86.9	85.7
Ordered	87.8	88.0	87.5	87.3	87.0
Invariant	88.1	88.2	88.0	87.7	87.2