FedDrop: Trajectory-weighted Dropout for Efficient Federated Learning
Authors: Dongping Liao
Xitong Gao
Yiren Zhao
Hao Dai
Li Li
Kafeng Wang
Kejiang Ye
Yang Wang
Cheng-zhong Xu
Venue: ICLR 2022 Submitted (2022)
Keywords: federated learning
efficient training
dropout
stochastic model
channel selection

Abstract: Federated learning (FL) enables edge clients to train collaboratively while preserving individual's data privacy.  As clients do not inherently share identical data distributions, they may disagree in the direction of parameter updates, resulting in high compute and communication costs in comparison to centralized learning.  Recent advances in FL focus on reducing data transmission during training; yet they neglected the increase of computational cost that dwarfs the merit of reduced communication.  To this end, we propose FedDrop, which introduces channel-wise weighted dropout layers between convolutions to accelerate training while minimizing their impact on convergence.  Empirical results show that FedDrop can drastically reduce the amount of FLOPs required for training with a small increase in communication, and push the Pareto frontier of communication/computation trade-off further than competing FL algorithms.

Under review as a conference paper at ICLR 2022
FEDDROP: TRAJECTORY-WEIGHTED DROPOUT FOR EFFICIENT FEDERATED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning (FL) enables edge clients to train collaboratively while pre- serving individual's data privacy. As clients do not inherently share identical data distributions, they may disagree in the direction of parameter updates, resulting in high compute and communication costs in comparison to centralized learning. Recent advances in FL focus reducing data transmission during training; yet they neglected the increase of computational cost that dwarfs the merit of reduced communication. To this end, we propose FedDrop, which introduces channel- wise weighted dropout layers between convolutions to accelerate training while min- imizing their impact convergence. Empirical results show that FedDrop can drastically reduce the amount of FLOPs required for training with a small increase in communication, and push the Pareto frontier of communication/comptaaiii trade-off further than competing FL algorithms.
INTRODUCTION
In the light of the importance of personal data and the recent striet privacy regulations, e.g. the General Data Protection Regulation (GDPR) of the European Union (Voigt & Von dem (asssche/[217) Wolters
2017 Politou et al. 2018). there is now a great amount of risk, responsibility (Edwards al. 2016
Culnan & Williams 2009) and technical challenges for securing private data centrally (Sun et al.
2014); it often impractical upload, store and use data central servers. To this end, federated learning (FL) (McMahan et al., 2017: Li et al. 2019a) enables multiple edge compute devices to learn a global shared model collaboratively in a communication-effiiee way without collecting their local training data. When compared against naïve decentralized SGD, Federated averaging (FedAvg) (McMahan al. (017) and subsequent FL algorithms (Li et al. 2020 Karimireddy et al., 2020) reduce the burden of data transmission by many orders of magnitude.
While these communication-effiienn methods can notably alleviate the difficulties of using FL in scenarios with limited bandwidth or data-quota, they, however, entail a drastic increase in computation cost, which has rarely been addressed by previous literature on FL. It has been argued that communi- cation is several orders of magnitude more expensive than computation 2019a; Huang et al. 2013). Yet existing FL methods optimize for communication so aggressively that an iPhone 12 Pro running as FedAvg client in tes would spend at least 16 minutes training, before even transmitting 180 MB of data, which only takes fraction of a second under the modern 5G infrastructure. The savings in communication thus are completely dwarfed by the expensive computation.
edge devices (Li et al.)
In FL. setup, models trained by the clients may disagree on the gradient direction to update the same neurons if the user data distributions are non-IID. In Figure[[] we demonstrate this effect with an initial round of FedAvg training. turns out that clients sharing similar data distributions tend to produce similar update trajectories to the same channel, and different distributions observe disparate trajectories. From this example, it can be observed that the consequence of non-IID data is two-fold. First, a naïve averaging of client parameters may cause these conflicting signals to cancel out each other, resulting in a slow convergence of the global model. Second, neurons layer tend to learn distinct features, and yet they cannot learn meaningfully if these features are absent in the client's s training data. Since neuron training is heavily dependent on the client's 's local data, presents us an
See Appendix P.6 for the setup.

---

Under review as a conference paper at ICLR 2022
+
Figure I: We synthesized 20 clients with the Fashion- MNNIS training data, where every two clients received only the same class images to simulate a concept shift, and separately trained the same CNN model for epoch. We present all filter parameters update magnitudes for the first 8 channels (row) in the first convolutional layer for all clients (column) after round training. The pairs of clients grouped together with same colors (one lighter and one darker) signify that they shared the same data distribution. Each polar plot contains the update magnitudes of all parameters the same channel. The length of each ray is the magnitude of the parameter update, and the parameters are arranged radially in each plot.
opportunity: can we concentrate training effort neurons that are correlated the current data distribution of the client, while paying less attention to neurons that are less relevant to the client? To leverage this, we introduce FedDrop, which introduces SyneDrop layers structurally sparsify client models, and the server-side an inter-client trajectory- -based optimization of the dropout probabilities used by SyneDrop to speed up model training. FedDrop brings two-fold advantages: dropped neurons can simply be skipped, and thus reduce the training FLOPSs required per step; and dropout probabilities each neuron can be tuned individually to minimize the impact on the global averaged model updates to assist convergence. Overall, as evinced by our experiments, FedDrop enables a much improved communication/coppaaiion trade-off relationship than traditional FL approaches.
Our contributions in this paper are as follows:
We present SyncDrop layers, which are synchronous structural dropouts with adaptive keep probabilities, to reduce the computational cost required by FL.
With SyncDrop. we formally derive the FedDrop objective that adjusts each neuron's dropout probability to improve convergence by minimizing the disparities among inter-client update trajectories. We further introduce FLOPs-based constraints enforce sparsity per FL round, allowing the trade-off between FLOPs and communication to be tuned easily.
Empirical results reveal that the combined method, FedDrop, attains a substantially better communication/cooputttii trade-off in comparison to other FL methods.
RELATED WORK
Federated learning. Distributed machine learning has a long history of progress and success (Peteiro Barral & Guijarro-Berie Berdiñas 2013 Li al., (0044), yet mainly focuses on training with IID data. The Federated Learning (FL. paradigm and the Federated Averaging algorithm (FedAvg) initially introduced by McMMhan al. (2117) allow clients to train collaboratively without sharing the private data in communication--ffiiiii
manner, To further tackle data heterogeneity. FedProx (Li et al. 2020 introduces new regularizations, and SCAFFOLD (Karimireddy et al. 2020) presents control variates to account for client drifts and reduce the inter-client variance. W While being effective at reducing communication, the above methods neglected the computational costs associated with the training process.
2

---

Under review as a conference paper at ICLR 2022
Computation vs. communication during training. There are few precursory methods that focus on the joint optimization of computation and communication costs during training. Caldass et al. (2018a) introduced federated dropout, which prunes parameters following uniform random distribu- tion, whereas PruneFI. (Jiang et al. 2019) proposes greedy wradient-mannitude-- pruning. In each FL. round, both methods produce shared pruned model with lineggriieed spar- sity for all clients. Such unstructured sparsity is difficult to utilize for training acceleration, and shared global model cannot exploit the data distribution of individual clients. Adaptive federated dropout [Bouacida et al. 2020) partially addresses the latter issue by allowing each client to select sub-model to join training. FjORD (Horvath al. 2021) introduces ordered dropout tackle the problem of system heterogeneity in FL with sub-model training that dynamically changes model capabilities using uniform sampling. FedGKT (He al. 2020) transmits shallow layer activations to offload training of subsequent layers to the server. However, the privacy implications were not well explored, and the trained models were substantially larger, which may limit their inference speed on edge devices. It is noteworthy that FedDrop differs from these approaches as it takes into account the non-IlI client data distributions in FL setting, and computes inter-client channel selection decisions that would minimize the impact convergence.
unstructured
Dropout algorithms in centralized training. Dropout (Hinton et al. 2012 Srivastava et al. 2014) may improve neural network training by reducing the overfitting effect and allow it to generalize better. done by randomly setting parts of the connections or weights to zero in a network, and scaling the remaining values accordingly. Since its advent, there have been an increasing interest in applying dropout in a structural fashion (Huang et al. 20155] Ghiasi et al. 2018 Hou & Wang (2019). Nonetheless, the above methods missed the opportunity explore the implications structural dropout on the FLOPs consumed by training or inference.
Structural pruning. A closely related topic is structural neuron pruning/selection for accelerated inference. These work propose new ways to extract a smaller accurate network from the original (He) al., 2018 Wu al. 2019 Li et al. 2019h Herrmann al. 2020p, and can even do so dynamically with an input-dependent policy (Gao et al. 2019 Hua et al.
2019 Wang et al. (020c).
THE FEDDROP METHOD
3.1
HIGH-LEVEL OVERVIEW
[a]
FLOPs budget
Last round
(c)
Parametor Update Trajectories
Inter-Client
Optimized
Trajectory Similarity
Dropout Keep Probabilities
Cient
Optimize
Client 2
0
Cient
GWYY9
Client 3
0
Clinn
siarita
chamnel
Client
Training round
Weightad average madel parameters
Training round
Server aggregation round
r r++
Figure 2: A high-level overview of FedDrop using FedAvg as the base algorithm. Here, we use 4 training clients for illustration. During each round of server averaging, FedDrop carries out dropout probability optimization to encourage collaboration between sparse clients for the next training round. Intuitively, for each neuron, if clients agree upon an update direction they would use larger keep probabilities, and disagreement results in lower probabilities. Clients thus focus training effort to their specialized neurons.
FedDrop complements existing FL methods by adding new client- and server-side components. During client training, convolutional layers are sparsified by interleaving them with channel-wise dropout layers, namely the SyncDrop layers. Section 3.3 discusses in depth how SyncDrop layers compute dropout decisions. Additional optimization stages during server aggregation encourage collaboration between sparse clients. Figure shows how FedDrop extends traditional FedAvg (McMahan al. 2017). After each round of client training, the server begins by identifying
added to
3

---

Under review as a conference paper at ICLR 2022
parameter update directions of each client from the previous round, and computes cross-client trajectory similarity matrix for each channel neuron. This is then followed by an optimization stage that iteratively minimizes the trajectory disparities by tuning dropout keep probabilities for each channel neuron on each client for the next training round (Section 3.6). This process also takes into consideration the FLOPss budget constraints sparsify client models (Section (.4). Finally, following FedAvg, the server broadcasts the weighted-average model parameters to all training clients in the new round, and finally sends the optimized probabilities the respective clients.
3.2 PRELIMINARIES AND DEFINITIONS
FedDrop complements most FL. algorithms. In this paper, we focus on its use on FedAvg (McMahan ct al. 2017). We assume the training loss function of a client c E to be (o())), where 0. comprises the parameters ! : E L} of all layers in the model of client
In each FedAvg training round r, clients begin by training on the loss function, with initial parameters g(r) received from the server for this round:
0 (++1) SGD (Cc, 000) B).
(I)
Here SGD, indicates that client c carries out stochastic gradient descent (SGD) on (o())) locally, and it uses a learning rate yj for E' epochs. The FedAvg server then aggregates client model parameters after the ndh training round, by taking the weighted average of them:
g(++1) = Eeec (T))
(2)
where de is the weight of client and proportional to the size of its training set [Del with Exec de 1. Finally, the (r + 1)th training round starts by repeating the above procedure.
3.3 SYNCHRONIZED DROPOUT
To induce sparsity in a stochastic manner, and subsequently introduce training correlation across clients, we designed threshold-baat based dropout layer, SyncDrop, synchronize dropout decisions across multiple clients.
Initially for all sampled clients E C S C, the server provides them with their corresponding dropout keep probabilities p. E [0, 11 and we assume p E [0, 11 Cx to the concatenated probabilites from all sampled clients. Each element p in p denotes the keep probability of the channel neuron n E N in client с E C. We additionally use p!, a slice of P.. to indicate the probabilities of all channels in the 1h layer. During training, the 1hh layer samples t!l! from uniform distribution 2/(0, 1) for all channel neurons n in the layer, where t shared across clients by using the same random seed. If P t! a channel in client is dropped, i.e. otherwise, the channel values are scaled by 1/p" during training") Formally, for each client c, the 1hh dropout layer computes for the input x!);
the entire channel map to zero;
drop(x"", (oppxx...po .P!) A d o x o _ p where d A 1[t < P
Here, [z]] is the (element-wise) indicator function, which is equal to when the condition z is met and 0 otherwise, refers the element-wist product, the term p - represents the element- inverse of p!", and finally all elements of !!!l are independently sampled from 4(O0, 1) and shared across clients. Figure Ba/ provides a high-level overview of the procedure described above.
From the local perspective of a client c, the dropout decision d!d is equivalent to the Bernoulli distributions Bpp!). An important distinction from independent Bernoulli distributions is that the distribution of d!' is correlated across clients e C. Given a pair of clients (i. for channel n:
E [dd"]= f. 1[t p. 1lt p min min(p"".. P3))
(4)
This enables us to adjust the correlation of the same channel neurons between any pairs of clients, and Section 3.6 makes use of this property to minimize the model update disparities across all clients.
2This is known "inverted dropout" and implemented hy PyTorch (2021) and TensorFlow 2021 provides ablation results on scaling for dropout to justify the design choice.
Figure
4

---

Under review as a conference paper at ICLR 2022
000 :channel of different clients
dropped
SyncDrop
Newe prooomm
Syno Layer (na client communicaaions)
Ootoul scomdt
Channele (grouped dients)
client channel keep probsbilitie
SyncDrop
dropped client channel
dropout threshold, drawn from uniform distribution (1
Client
Client 2
Client 3
Clint
(a) The SyncDrop layer.
(b) Sparse convolution.
Figure 3: A high-level overview of the SyncDrop layer. (ab For each channel across all clients in forward pass, we draw ~ (0, 1) from the uniform distribution. If the keep probability (1 dropout probability) of a channel in client is less than u, then the channel is dropped from the model. Note that no inter-client communications are required, the clients synchronized by sharing an initial random seed. (b) When convolutional layer sandwiched between two SyncDrop layers, takes the advantage of both the imput- and output-side sparsities. As example, here dropping 50% of both input and output channels can result in a 4x FLOPs reduction.
define f to be the sparsified variant the original model f. where SyncDrop layers are placed immediately after each convolutional layer with ReLU activation (Figure 3b). This enables convolu- tional layers to doubly accelerated by taking advantage of sparsities at both ends and skipping dropped channels in input/output feature maps. Recalling the client model loss function les additionally use f. denote the accelerated variant of the sparse model with probabilities p.-
is noteworthy that active channels are sampled for each local training mini-batch. In addition, the synchronization of dropout decisions is carried out with identical random seeds for all clients, and thus eliminates the need to communicate between clients. Finally, for inference, one may choose to enable the SyncDrop layers for speed, or skip these layers entirely. In our evaluations, SyncDrop layers are disabled for improved test accuracies.
3.4 FLOPS CONSTRAINTS
To encourage sparsity in models and for the optimization p. we adopt a hyper- aaperr-prramter r (0, that adjusts the FLOPs budget ratio, and define a shared global FLOPP-budget constraint g 2 for all participating clients C _ one round, where:
g(p)=r- - - E.
flops(f. Dossso p.) flops())),
(5)
and the terms flops(f, Pe) and flops(f) respectively denote the FLOPs of a sparse model with probabilities P.. and the FLOPs of the dense model f. Moreover, this constraint can be easily modified to allow client- and layer-wise, and even heterogeneous budgets by summing over targets with finer granularity. In Appendix we explain how the FLOPs of a model is computed.
3.5 PARAMETER INITIALIZATION
At model initialization, we set all values in p to be a uniform constant p such that g(p) = -O. Since the training clients are sparse at initialization, the variance of each initialized parameters 0 E lll of the 7th layer must satisfy the following condition:
Theorem (Parameter initialization). To avoid vanishing/explodiin gradients at initialization, the variance of the parameters of the 1 layer followed by dropout layer with keep probability should be 2p/N, where = CP is the size of the receptive field of each output channel.
In this paper, we introduce a modified He initialization (He et al. 2015), that is NO, 2p/N In Appendix C.I we provide proof of this theorem. Both (Hendryccs & Gimpel (016) and Pretorius cet al. (018) derived similar schemes for fixing parameters imitialization of layers followed by dropout. Figure further justifies this decision empirically by ablation of the fixed initialization.
5

---

Under review as a conference paper at ICLR 2022
3.6 OPTIMIZING DROPOUT KEEP PROBABILITIES
In our experiments, we found that using a constant dropout keep probability is often detrimental to the joint optimization of computation and communication costs. To speed up training while minimizing the impact of model sparsity on global training convergence, we therefore aim to optimize the dropout keep probabilities p for the following objective:
minp E Volcdo Eoccc
(6)
Le. the mean square error between the averaged gradients of all training clients with and without SyncDrop layers. Intuitively, it aligns the parameter updates of sparse models with the original dense variants for an improved convergence behavior. As result of the correlated dropouts across clients
(4). the objective above can be minimized by adjusting all dropout keep probabilities p across neuron n on client c.
As it is impractical to communicate each step of SGD across clients, FedDrop instead optimizes an alternative objective derived in the theorem below (proved in Appendix C.2 that approximately minimizes the original:
Theorem 2 (Approximate Objective). Assuming that for channel neuron E N and pair of
clients i,j E C, we have Si; & Xıdy max(p;; DD) A A0%, where A0 Λ0;; are the param- eter updates after a round of FL training, D. Dj are the current dropout keep probabilities, and
max(x, y) represents the element-wint max between x and y, the optimization objective of (6) can be approximated by:
ming obj(š, q) ming EmeN. Sy, Stj/max(qa ,9))
The optimized values q can subsequently be used as the new dropout keep probabilities for the next round of FL training.
It is also desirable to have bounded optimization avoid extreme solutions. Without FLOPs constraint, both objectives (6) and (7) can be trivially minimized with 1, i.e. when none of the channels dropped, and Appendix C.2 provides the proofs of the following theorem:
Theorem 3 (Optimization bound). The objective ming Enis Sty max(qq. 9ī) is trivially minimized to Eni Sij when L = 1 for Vn E NN,EEC.
To enforce sparsity, we therefore further constrain the optimization of to be within the feasible set (P) 2 0 as defined in Section 3.4 In practice on the server-side after gathering model updates from round of sparse client training, we minimize the overall objective via gradient descent with the interior point method:
ming obj(š, q) y lo)(
The term - log(g(g)) constrains the solution q to be within the feasible set g(q) 2 0, and the hyper- parameter tunes the strength of the regularization, and is kept constant at 10-4, In Appendix D we describe the overall FedDrop algorithm depth, and discuss the computational and memory implications of FedDrop.
4 EXPERIMENTAL RESULTS
In this section, we present comprehensive evaluation of both the communication and computation costs of FedDrop and compare against other FL algorithms, namely FedAvg (McMahan al.) 2017), FedProx (Li et al. 2020). SCAFFOLD (Karimiredyy et al. 2020). and Caldas et al. (2018a) We also introduce UniDrop for reference - simple channel- dropout with a uniform keep
probability that is kept constant as the comparison bascline.
We conduct experiments on three popular open datasets CIFAR- 10 (Krizhevsky et al. 2014), Fashion- MNIST (Xiao al. 2017) and SVHN (Netzer et al. 2011). To simulate non-llD training data distribution with a concept shift, for every class label we split the training dataset into |C] parts to be respectively received by the clients, where the size of all parts follow the Dirichlet distribution Dicy (a) (Hsu et al. 2019 Wang et al. 2020b). For the full participation case, we used a = 0.5, 20 clients for CIFAR- 0, and TOU clients for Rashion-MNIST and SVHN in all experiments by default.

---

Under review as a conference paper at ICLR 2022
We used 9-layer VGG-style architecture (Simonyan & Zisserman 2013) for CIFAR-10, wheras Fashion-MNIST used a LeNet (Lecun et al. 1998) model and SVHN a smaller variant, and we provide the details of the models and datasets in Appendix
For a fair comparison, we performed a grid-based search on the remaining hyperparameterss for baseline algorithms find configurations that provide the best accuracy after 100 rounds of training for each dataset. The search results suggested that typically setting the batch size B = 4 and learning rate 7 0.02 were universally optimal for most methods under comparison. FedProx additionally introduced regularizer which was also optimized within {0.0001, 0.001, 0.01, 0.1 Finally,
FedDrop introduces global FLOPs budget per round as ratio between r (0, 1] that can be adjusted to steer the communication/come conppuutiioooooomutttttnon trade-off as defined in (5), for which we specify the values the experiments below.
FLOPs vs. Accuracy. We first focus on the amount of FLOPs required by the algorithms, and provide
comparisons across the FL methods for the same number of local epochs per round E, given their respective optimal hyperparameters with unbounded communication budget (Figure a). Here, we used a FLOPs ratio r 0.5 for FedDrop. Frequent communication rounds allow the server to more frequently optimize dropout keep probabilities for FedDrop, which can notably improve convergence under the same FLOPs budget. contrast, other methods struggle to improve their convergence speed.
80
Accuracy (%)
Accuracy (%6
60
Accuracy
Tedklo UnDop
FedDre
redbing
UnDkop
Felltrs
SCAFFOLD
SCAFTOLD
SCAPFOLD
Calticaal
Calla
0.4 0.6
1.2
0
3
0.5
1.5
Number of FLOPs
-1015
Number of FLOPs -10
Number of FLOPs -1044
(a) = 4.
(b) E = 2.
(c) 1.
Figure 4: Comparing FL methods with the same local training epochs per round F. used for CIFAR-10.
FLOPs vs. Communications. To bolster the trade-off between communication and computation, we additionally conducted a hyperparameter exploration to navigate the Pareto frontiers of this trade-off relationship. The hyperparameters include the number of local epochs per round F. and the batch size B. In addition, FedDrop can search for the Pareto optimal in the set (0.2, 0.25, 0.333, 0.5). notable that with
Figure 5 shows the FL.OP//communication trade-off of the methods. decreasing FLOPs budgets, FedDrop expends minimal communications, whereas other methods may have significantly larger communication costs to reach the same target accuracy. Conversely, in Table I show that FedDrop be up to 3x more efficient in terms of FLOPs under the same communication budget for the same accuracy goal.
109
Fetllng
10F
Fadiva
Communications (GB)
libbee
Communications (GBB
Du
Tabley
Falling
Communicaaions
101
Calbi
10a
10a
10a
1044
102822
10284
10-38
1013.
10134
1013.
10-3.
10
1044.
10-4.8
1015
1046.2
1015.
Number of FLOPs
Number of FLOPSs
Number of FLOPS
(a) 85% for SVHN.
(b) 85% for Fashion-MNIST.
(c) 80% for CIFAR-10.
Figure 5: Comparing the FLOPs vs. communication trade-off across different FL methods reaching a target accuracy for given dataset. We highlight the Pareto optimal points with dotted lines.
7

---

Under review as a conference paper at ICLR 2022
Table 1: Comparing the computational costs (in TFLOPs) required by FedDrop against other methods given a communications budget. Multipliers in parentheses signify the magnitude of FLOPs increase compared to FedDrop.
Methods
Dataset
Acc.
Comma
FodlDrp
UniDrop
McMalann et all, 2017
FodAva
Li al. 2020
FodPinx
SCAFFOLD Karimireldy et 2020
Caldas at 2018s
CIFAR-10
70%
16 GB
CB8
244.79
591.35
964.12
856.99 (3.00x)
602 3
569. (2.32x)
669.59 71
589 24 (2.41x)
696 37 (1.78x)
549.06 (2:24x)
622.48 (2.54x)
(15.19(1.85
80%
753.67
1526.52 (2.01x)
1208.88
800.55
1098.23
964.21 (1.81x)
1071.34
870.47 (1.64x)
1172.12
(1.76x)
(2.54xc)
(2.43kc)
(2.71
Fashion-
GBB
26.97 (2.74x)
(3.14()
30.54 (3.11))
26.99 (2.78kc)
31.65
MINSS
8546
GB
(BB
$2.83
25.69
65,30 (1.99x)
(2.32x)
66.77 (2.60xx
08.19:12.08:
08.19 (2.08x)
62.51 (2.43x)
50.83 (1.73x0
$1.14 11.14(1..00 (1.9900
72.02 (2.90x)
94.78 (2.87x)
SVHN
8056
GBB
GB
6.45
7.72
13,08 (2,02x)
(1.77))
12:25 (1.59%)
11.41 (1.77x)
11.42 (1.77x)
12.25 (1.59x)
12.22 (1.58
11.70 (1.81
13,48 (2.09x)
(2.24))
8556
GB
12.23
13.30
(1.91x)
26.60.2.00x0
(1.82.1144x))
19.99
17,82
19.99 (1.50x)
22.19 (1.67
16:15
43,85 (3.30x)
(2.30x)
Fractional Device Participation. To measure the performance of FedDrop in more practical dis- tributed scenario, we scale the number of devices to 1000, while reducing the fraction of participation rate é per round only 1 1,, yielding results in Table We refer readers to Appendix F.5 for more detailed trade-off results.
Table 2:
Comparing the computational costs (in TFLOPs) required by FedDrop against other methods given a communications budget with 1000 devices and 1% device participation ratio (ø 0.01).
FelDrop
UniDaop
McMalun et 2017
Foddyp
2000
CIFAR-10
70%
320 (RR
160 (38
122.74
99.80
290.98
213.96 2.14
(2.57x)
215.07 (1.25k)
194.50 (1.95x)
213.46
187. (1.88)
238.93 (2.59%)
469.64 (3.85x)
75%
320 (IR
150 GB
223.28
128.52
392.79
319.57
(1.76x)
(2.49))
233.15
522.47 (1.46x)
225.55 (1.75x)
321.13 (1.4x)
(3.299)
(3.09))
GB
1.32
1.89 (1.43x)
2.70 (2.05k)))))
(1,96))
(2.9
Fashion-
GB
1.04
(2.16x)))
(2.23x)
MNIST
(1.39)
(1.24x)
GB
(2.022)
5.54 (2.01x)
(1.97%))
(2.55%)
SVHN
S056
GB
4GB
0.66
0.56
0.99 (1.77X)
1.11 (1.68)
0.95 (1.20xx)
L.05 (1.59x)
8.9 (1/02X)
1.04 (1.58x)
1.14 (2.84))
1.39 (2.11))
83%
4 GB
GGB
0.94
1.36
2.17 (2.38))
2,34 (1.72))))))
L.65 (1.21x
1.40 (1.499))
1.37 (1,46)
(1.188)
2,50 (2.66x)
(2.282.288))
Dataset Distribution Imbalance. To investigate how different data distributions affect the perfor- mance of FedDrop rigorously, provide sweep of a E (5, 0.5, 0.05) for the CIFAR- 10 dataset splits Dicj(e (a) to simulate increasing degrees of distribution imbalance in Figure 6 shows that FedDroop is the best method at maintaining accuracy with distribution imbalance w. the FL.PPs budget. We believe SCAFFOLD may be unsuited for larger models as diverged under these scenarios.
80
80
60
Accuracy (%)
60
60
Kalla
Accurncy
FelDres Cuiree
Accuracy
40
PaDlp Unttap
40
Unbbep
40
20
SCAFROLD
20
SCAPROUD
20
SCAFROLD
0.5
1.5
2
0.5
1.5
0
0.5
1.5
Number of FLOPss
-1045
Number of FL.OPs -1015
Number of FLOPS -1045
(a) F = 8, a 5.
(b) E = 8, = 0.5.
(c) E = 8, 0.05.
Figure 6: Varying a for the dataset splits D (a) of CIFAR-10 with = 8.
Ablation Study. To verify our design choice, we conduct a thorough set of ablations in Figure 7 We observe certain degree of accuracy drop as these components are ablated, confirming the significance
8

---

Under review as a conference paper at ICLR 2022
80
80
250
(96)
-70
70
60
20
50
30
L.food
60
40
40
40
Number of Communication Rounds
Number of Communication Rounds
Number Communication Rounds
Number of Communicaiion Rounds
(a) r= 0.5
(b) r= 0.25
(c) r 0.5
(d) r= 0.25
Figure 7: Ablation of design choices. (a. b) SyncDrop with and without synchronization. (c, d) with and without dropout scaling and fixed initialization under different FLOPS ratio for CIFAR-10.
of synchronization. This then further justifies the effectiveness of dropout scaling and initialization that comes with theoretical guarantees.
AASTIRO (%)
Accumcy
75
Accumio
75
Accoumicy
70
OBI
ETS
00
20
40
3U
100
20
60
60
80
100
10
10
10
10146
10
201
113
60
80
Nomber of Communication Rounds
Number of Communication Rounds
Nomber FLOPs
Number of Communication Rounds
(a) Optimization iterations.
(b) FLOPs regularizer.
(c) FLOPS vs. accuracy.
(d) Rounds accuracy.
Figure 8: Sensitivity analyses of (a) optimization iterations 1, (b) FLOPs constraint regularizer μa, and (c, d) the FLOPs budget ratio r.
Sensitivity Analysis. Figure produces a sensitivity analysis of the hyperparameters introduced by our server-side probability objective minimization 7 namely the number of gradient descent iterations and the strength of regularization μu. turns out that additional iterations can improve convergence, which is desirable the server compute budget is usually not the performance bot- tleneck. Moreover, a broad range of regularization fJ do not have an adverse impact on accuracy, eliminating the need for search. We also present analysis the effect of the FLOPS ratio as defined in 5 Reducing this ratio can sparsify models, for faster initial convergence w.r.f. FLOPP. However, comes a cost of slower convergence latter rounds.
Additional Results. In Appendix F we provide additional results and analyses. Specifically, Ap- pendix F.I includes the trade-off curves for all datasets under different accuracy budgets. We examine the performance impact on FL algorithms with different degrees of data imbalance (Appendix F.2). and varying hyperparameters such local epochs per round (Appendix) F3 and the FLOPs budget ratio r (Appendix F4) Finally. Appendix F.3 provides full results with fractional device participation.
5 CONCLUSION
Recent federate learning (FL) algorithms focus on the reduction of communication costs, while neglecting the expensive local compute required by edge clients. This could be further exacerbated as the models we employ may increase in size over time to attain higher task performance. presented FedDrop, novel FL technique that uses synchronous dropout with adaptive keep probabilities to concentrate the clients' training effort neurons that they specialize well, while making the models sparse to reduce computational costs. Our experiments show that FedDrop can push the Pareto frontiers of communicaiion/compttiiii than existing algorithms. We believe this paves the way for future work that allow FL algorithms to scale up models being trained considerably, and consequently improve their performance more challenging training tasks.
trade-off of FL scenarios notably further

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
We believe FedDrop can have a positive impact on the environment as aims to make federated learning more efficient.
The federated learning algorithms require communicating locally optimized client models to global server. In such scenario, training data could potentially be recovered from model updates (Yin et al. 2021; Wang al.) (020a), and this could have privacy implications. Although our work do not empirically address the privacy concern, believe the added dropout layers do not bring noticeable impact to the underlying FL. algorithm, for the reasons below:
The dropout probabilities are computed on the server with client updates, the optimized probabilities simply derived from information that already collected by the base algorithm.
There is no inter-client information exchange or leakage, as each client only receives its corresponding dropout probabilities.
REPRODUCIBILITY STATEMENT
We provide detailed information about model configuration in Appendix D dataset splitting and partitioning in Section 4 and hyperparameters for optimization in Section We try to justify the design choices of our method with ablation in Section 4 and all the experiments reported in this manuscript are conducted exhaustively with broad choices of hyperparameters to ensure fair evaluation for all methods. The code would be available during the review period and made public upon acceptance.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Core MI. - integrate machine learning models into your app. Technical report, Apple Inc., 2021.
Nader Bouacida, Jiahui Hou, Hui Zang, and Xin Liu. Adaptive federated dropout: Improving communication efficiency and generalization for federated learning. CoRR, abs/201 0..20011.04050 .440500 2020.
URL httpss///arxiivoog////////1 1.040000 .04050.
Sebastian Caldas, Jakub Konečny, Brendan McMahan, and Ameet Talwalkar, Expanding the reach of federated learning by reducing client resource requirements. arXiv preprint arXiv:1812.072I11
arXiv:1812.07210, 2018a.
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konecný, H. Brendan McMahan, Virginia Smith, and Ameet Talwalkar. LEAF; A benchmark for federated settings. ArXiv, abs/1812.01097, 2018b.
http://arxivvorr/abs////111000
Mary Culnan and Cynthia Clark Williams. How ethics can enhance organizational privacy: lessons
from the choicepoint and TJX data breaches. Mis Quarterly, pp. 673 673-687, 2009.
Benjamin Edwards, Steven Hofmeyr, and Stephanie Forrest. Hype and heavy tails: A closer look at data breaches. Journal of Cybersecurity, 2(1):3-14, 2016.
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng zhong Xu. Dynamic channel pruning: Feature boosting and suppression. In International Conference Learning Representa- tions, 2019. URL
Golnaz Ghiasi, Tsung- Yi Lin, and Quoc V Le. DropBlock: A regularization method for convolutional networks. In Advances in Neural Information Processing Systems, volume 31, 2018. URL https:/prooceedings.nu wwwp:/nng..nuuuicsc/ossseeee
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. http://proceedings..lle http://ooeeednnesimipprrr//////s www.o
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. In Advances in Neural Information Processing Systems 33, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Sur- passing human-level performance on imagenet classification. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV '15, pp. 1026-1034, USA, 2015. IEEE Computer Society. ISBN 9781467383912. doi: 10. 10.11000..........2.3.. URL http://doi.org/I00 10. 1091111....2.21.......
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. AMC: AutoML for model compression and acceleration on mobile devices. In European Conference on Computer Vision (ECCV), 2018
Dan Hendrycks and Kevin Gimpel. Generalizing and improving weight initialization. CoRR, abs/1607.0244 024888 2016. URL http:///arxiv.orr/bbb//1107 02488
Charles Herrmann, Richard Strong Bowen, and Ramin Zabih. Channel selection using Gumbel softmax. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Euro- pean Conference on Computer Vision (ECCV), pp. 241-257, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58583-9
Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. CoRR, abs/1207 11.01100788880 0580, 2012. U http:://aaxiiiorr/aab///aaaaaaaa
11

---

Under review as a conference paper at ICLR 2022
Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos I. Venieris, and Nicholas D. Lane. FjORD: Fair and accurate federated learning under heterogennous targets with ordered dropout. CoRR, abs/2102.13451, 2021. URL http://axxi.orrr//s2211...115
Saihui Hou and Zilei Wang. Weighted channel dropout for regularization of deep convolutional neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8425-8432, 2019.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification, NeurIPS Workshop on Federated Learning for Data Privacy and Confidentiality, abs//909.0000 009.06633, 2019.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Federated visual classification with real-world data distribution. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16, pp. 76-92. Springer, 2020.
Weizhe Hua, Yuan Zhou, Christopher M De Sa, Zhiru Zhang, and G. Edward Suh. Chan- nel gating neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL http://moeecconnss.eerrcpcccccop (tw.lbe777oc/aa0000949111222066 Papppp.
Junxian Huang, Feng Qian, Yihua Guo, Yuanyuan Zhou, Qiang Xu, Z. Morley Mao, Subhabrata Sen, and Oliver Spatscheck. An in-depth study of LTTE: Effect of network protocol and application behavior on performance. In ACM SIGCOMM 2013, pp. 363-374, 2013.
Yuchi Huang, Xiuyu Sun, Ming Lu, and Ming Xu. Channel-ma max, channel-drop and stochastic max-pooling. In 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 9-17, 2015. doi: 10.1 0011099CVPPW22055500000...
Yuang Jiang, Shiqiang Wang, Bong-Jun Ko, Wei-Han Lee, and Leandros Tassiulas. Model pruning enables efficient federated learning on edge devices. CoRR, 0tbs/1999.123220 2019. URL /htp.
//arxiv.org/ab//1190 12326
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5132-5143. PMLR, 13-18 Jul 2020. URL http::/prooceeiings... mir.press/v 19/kriimiredyy 20a.edddyy0mmmm
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 and CIFAR-100 datasets. http://wwww.c..orroot.ooo edu/ kriz/eifar.hmml, 2014.
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. In Proceedings of the IEEE, pp. 2278-2324, 1998.
Mu Li, David G Andersen, Alexander J Smola, and Kai Yu. Communication efficient distributed machine learning with the parameter server. Advances in Neural Information Processing Systems, 27:19-27, 2014.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. ArXiv, abs/1908. 0008077773, 2019a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In I. Dhillon, D. Papailiopoolos, and V. Sze (eds.), Proceedings of Machine Learning and Systems, volume 2, pp. 429-450, 2020. URL had.eaaaaaaa....oooooooooeeeeed. Pappee
Yawei Li. Shuhang Gu, Lue Van Gool, and Radu Timofte. Learning filter basis for convolutional neural network compression. In Proceedings of the IEEE International Conference on Computer Vision, 2019b.
12

---

Under review as a conference paper at ICLR 2022
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication efficceen learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273 1183, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL httt:::/ufddd.sstatoodee.. www.fordd.duuoouueuuubeess
nips2011 hoousooumeesss.ppoe.
Diego Sheeroo-Barrl and Bertha Muujarro-Berdiian A survey of methods for distributed machine learning. Progress Artificial Intelligence, 2(1):1-11, 2013.
Fugenia Politou, Efthimios Alepis, and Constantinos Patsakis. Forgetting personal data and revoking consent under the GDPR: Challenges and proposed solutions. Journal of Cybersecurity, 4(1), 2018.
Arnu Pretorius, Elan van Biljon, Steve Kroon, and Herman Kamper. Critical initialisation for deep signal propagation in noisy rectifier neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL http://pwccccdnnnss..uurrs..///e
PyTorch. Pytorch documentation torch.rr.....c.... www..ch....uuciioona..rrr..ppppp 2021. URL httpss//pptoorch.orrrdoss
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15556):122--1955 2014. URL http:p//jmml...ggpppppss/// hatp://rrrrrrrppaes/s/ssvsaaaaa 4a.html
15/srivastaval
Yunchuan Sun, Junsheng Zhang. Yongping Xiong, and Guangyu Zhu. Data security and privacy in cloud computing. International Journal of Distributed Sensor Networks, 19070, 2014.
TensorFlow. Tensorflow documentation dr.nnn,r 2021, URL http:://wwww.eenssrfooooogg
docs/fythoontt////rrropuuu
Paul Voigt and Axel Von dem Bussche. The EU general data protection regulation (GDPR), A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10:3152676, 2017.
Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning, arXiv preprint arXiv:2007.05084 2020a.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Fed- crated learning with matched averaging. In Intermational Conference on Learning Representations, 2020b. URL
Yulong Wang, Xiaolu Zhang, Xiaolin Hu, Bo Zhang, and Hang Su. Dynamic network pruning with interpretable layerwise channel selection. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04)):629 Apr. 2020c. doi: (t.0///aai.34404..... URI. https://ts////os..aa org/index. htgpi/eeeppppsAAAllaaateeeeev//o
PTJ Wolters. The security of personal data under the GDPR: a harmonized duty or a shared responsibility? International Data Privacy Law, 7(3):165-178, 2017.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuan- dong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. FBNet: Hardware-aware effi- cient convnet design via differentiable neural architecture search. In IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2019, pp. 10734-10742. Com- puter Vision Foundation / IEEE. 2019. doi: -1009//VP..201.000000 peneenacess..hecc..coomcco SoawNet_Designe witee_iaaaaaeeeiiaileeeeeeeeeeee NewraalArrhhhtteeeeeeeeeee Sew.cc_CCaeaeaa_____ 2012_2019___0 2019_paperrhtmll
URL http:
CVPR 2019/html/Wu FBNet Hardware- Aware Efficient.
13

---

Under review as a conference paper at ICLR 2022
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image dataset for bench- marking machine learning algorithms, 2017. http:://ghhilccoo///aaoooeeeeees mnist
Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16337-16346, 2021.
14

---

Under review as a conference paper at ICLR 2022
A TABLE OF NOTATIONS
For your convenience, we provide list of notations used in this paper in Table
Table 3: Summary of notations.
Symbol
Description
an individual client.
C
set of clients.
C
set of participating clients.
66())
training loss function of client c.
B
mini-batch size.
De
the local training dataset of client c. the parameters of all layers in the model of client softmax- cetmmaxcrrosseetroope entropy (SCE) loss function. the feed-forward model.
0
fü
1th layer of model f.
L
the total number of layers of model f.
x
the model input.
the model output.
0_" g(++1)
the parameters of client e in round r + learning rate. epochs of local training.
E
the FLOPs budget ratio as defined in Section 3.4 (0, 1].
the fraction of participating clients, (0, 1].
A
weight of client с proportional to the size of its training data set [De]] (matrix) dropout keep probabilities.
p,q
Pe
(vector) dropout keep probabilities of client c.
(vector) a slice of pe indicating the probabilities all channels in the 1th layer. (scalar) the dropout keep probability of neuron n in client
P
t
thresholding vector for tt layer with n neurons. the (element-wise indicator function.
1/2]
the element-wiss inverse of binary dropout decision vector of 7th layer. sparsified model.
dg
g(.)
budget constraint function. loss function of client c.
fe
((++))
(tensor) trajectory similarity with shape |N| for the (r + 1) round.
x
CC| |C|
as in Figure bb)
S (r+1)
(scalar) trajectory similarity of neuron n between pairs of clients (i,j) the (r+ + 1)th round.
S
(tensor) estimated trajectory similarity accounting for the dropout variances.
flattened array of weight parameters of 1th layer.
Dic (ax)
Dirichlet distribution parameterized by a and |C].
T
positive semidefinite matrix. a permutation matrix.
II
h
boolean vector.
H-))
the Hessian matrix w.s.. parameters.
B COMPUTING THE NUMBER OF FLOPS
In sparse layer fil,, the expected number of FLOPs per image-step is the sum of the flops required by both the convolution and ReLU activation:
flops(f") 2(( tn " "WO H'"),,
15

---

Under review as a conference paper at ICLR 2022
where K2 is the 2-dimensional kernel size, and H x W is the output feature map size. Moreover, ((--1) and (t9ll are the number of active input and output channels respectively; we expect (!P), the average number of remaining output channels for layer 1, to be C mean [P mean where mean(z) takes the mean of elements in and C is the number of total output channels of layer I.
To generalize, can rewrite the total number of FL.OPs required by the overall model f to be:
flops( (f P.) = 1 mean(p") m (k Ct mean (p ()= + 1) H""
(10)
and assume pd = and p 1, since both the input and output of the model are considered to be dense. Finally, by setting p for the entire model, we get the number of FLOPs consumed by the overall model per image.
C PROOFS
C.I MODEL PARAMETER INITIALIZATION
Here, restate Theorem 1 from Section 3.5
Theorem (Parameter initialization). To avoid vanishing/exploding gradients at initialization, the variance of the parameters of the 1th layer layer followed by dropout layer with keep probability p should be 2p/N, where = CU- is the size of the receptive field of each output channel.
The theorem proof depends on the following definitions and assumptions.
Definition 1. A convolutional layer with channel dropout decisions d can be written
x++1] _ relu(z!)). where z - P
p1o-1 o d (1 +b")),
(11)
where we flatten the input features and the convolutional weights be respectively 2 x TR. Nx1
and 9 Ro x with N Kw.² Ct being the size of the input receptive field. Recall that indicates the element-wise product, and the notation 2 is the element-wise inverse of z. Here each dropout decision in d is sampled with:
a A 1[t < <p"".
(12)
At initialization, assume that pd shares a constant dropout keep probability p, and we can immediately derive that for any channel neuron i at initialization:
E(d,"] = p- + (1 p) var(d!") Eld p
(13)
-0 =
(14)
We assume the model input x! and for each layer I the intermediate values z to be and all have zero-valued means. We also let the individual weight parameters 9 to initialized with IID and zero means, and the biases bidl 0. Therefore, for each layer E [1, L):
Assumption 1.
E(z!)) = 0 for z!i E z E(O for 0 Mm.
(15)
(16)
To avoid vumishing/exploodngg gradients, we restrict the variances of each value in " be constant across all layers E [1, L}, i.c.
Assumption 2.
var(z)"") ( = var(z))") for m. E [1, z| [m)) 2)1 !n!
(17)
We adapt the proofs from (He et al. 2015) Glorot & Bengio 2010 by taking into consideration the variance introduced by dropout layers:
16

---

Under review as a conference paper at ICLR 2022
Proof. By Definition | for any value z! E 2 at initialization:
var(z.) var d oy xy d d
As d 0 and xf] are independent, and b at initialization following Assumption
N var (var(d.))
+ E[d!"))) (var() E[ (var(x,") E(x]
E
( [1 E
(18)
Again following Assumption var(d = p(1 - p):
E [ ky
Also, var(x)") + E|x E(x!") E(d.") = p. and
N var 00 E(x.!i))
var(z; (z. (I)
(19)
P
As proven by (He et al. 2015 for ReLU-nonlinearity, E(x]" = var(z)- ). and using Assump-
var(z)) "), and solve 19 to give var( - Finally, if we are using Gaussian distributions to initialize model parameters, then 0! ~ N(O, 2m)
tion 2 can let var (z.
C.2
OPTIMIZATION OBJECTIVE
This section provides proof sketches for the optimization objective 7 described in Section 3.6 Let's assume 9. - > Voles and 9. & Vo to simplify the derivations below.
Lemma 1. The sample gradient with dropout 9. can be approximated by de P. gg-
Proof. Consider the 1 1ddorder Taylor expansion of (0):
(0.) (d. p 0.)) (0.) + (Ve. (oo "p 00. -0) Differentiating both sides by Oc. we have the following, where H(e.) signify the Hessian of (0.):
Va Va le Vo. (Vo. L,) (d. o p 1- 1-1)00
= (V (d pe -1)
1) + (dd o D - 1) o (.) (((.).
We omit the second term ((d p.- 1 - 1) 0.) TH((.) for simplicity, as the Hessian of t. (0.) is compute-intensive, thus:
- ge = Ao (Vo voll) ~ ge (d. - -1).
(20)
In practice, we cannot obtain S E 9" as evaluating the gradients of dense models defeats the intention of accelerated training. We can instead estimate the value with the following lemma:
Lemma E E[g. g²) can be approximated with E[g. 97] max(p", p.), where p p.. are the dropout keep probabilities from the previous round of the neuron respective clients i,
Proof. From Lemma by rearranging
20
we have ge R 9e d o Pe for client c. Therefore.
E
R
P.P.
As d, and g. are independent for each client c,
= [g 9;
E P.P.
By the definition of the SyncDrop layer Section
3.3
[ f [[<p""""(<pp) di E
P.P."
E [9]
min(p" PP)) P.P.
Rearrange to give:
E [9] 9
E E[g 9 max(p". p.)
(21)
n
17

---

Under review as a conference paper at ICLR 2022
Finally, reiterate Theorem 2 that introduces the approximate objective 7
Theorem 2 (Approximate Objective). Assuming that for channel neuron E N and pair of clients i,j E C, we have S X:A3 , p)) A A07. where 20% A0; are the param- eter updates after round of FL. training, P. P are the curvent dropout keep probabilities, and
max(x, y) represents the element-wit max between x and y, the optimization objective of (6) can be approximated by: ming obj(S. q) a ming E EmeNaJec Sij/max(qq 99))
Proof. We start by deriving from 66:
minp E
Ec Ececdo Vo
[E (Ececle- - 9.))
( 9
following Lemma
1 EiJeCC,eN E o0 1) 1)
Eijec E (0. (d. of -1) (g, (d, (d)
as de and g. are independent for each client c, and following Lemma 2
EijjeC,nen [ max(p?) pJ)
d'
1) d'
1)}
by definition of the SyncDrop layer in Section 3.3
EiJjEC.NeN [[ max(p? p3) So ("
1)
[[(59 1)dt
q
95
EijecC,en E [ max(p,;, p
(22)
q." q..)
Finally, as it is impractical use single step gradients O and 93 on the server, we approximate them with the respective parameter updates after one round of training, i.e. \0 and X, AyA0f; recall the definition of S, we have
ΣijjcC,en max 9. 97)
Eijjec,ner' Siy-
(23)
The first term is exactly 7h, and the second term is constant and can be safely removed from the optimization objective.
The value Sy can be considered the similarity of the model updates between clients i and for the group neuron n. Intuitively, from the perspective of an individual neuron n and the two clients and minimizing the term S /max(p," P3) incentivizes p. and p to grow when S is positive, and to become smaller when is negative.
With the derivations above, can proceed to prove the objective in Theorem is bounded with minimum. First, we can show that the matrix S" is positive semidefinite for each neuron n E Lemma 3. The matrix SD is positive semidefinite for n E N.
Proof. By definition, S = E[g"g" g" where g" comprises the gradient vectors of the nh neuron across clients C. The matrix St is therefore positive semidefinite, it is averaged sum of samples of positive semidefinite matrices "T
Theorem 3 (Optimization bound). The objective ming Eney $ ///x/x/xxf/ ag) is trivially minimized Enig $7 when qid 1 for Vn E N. E C.
18

---

It is notable that 2 1 for all c E C. The first term is bounded by a non-negative 1T i.e. the sum of all elements in positive semidefinite matrix. The second term is also non-negative, as by the ordering proposed in (24). Notably if π. = 1 for all n E N and c C, then the objective is trivially Under review as a conference paper at ICLR 2022
Proof. Without loss of generality, can permute the matrix S" and the probability vector p" to assume a descending order on the probabilities. Let T" II and π ПII" denote the respective permuted trajectory similarity and probability variants, where II E -00 1] [c]][]| is the permutation matrix, such that
$
Emj (p DY)
Σ nij m and .
π [c]
(24)
We introduce a boolean vector h" E RRC where each element h of it is defined as:
h 2 1 T 2 T:
we then let H & h"h²T, and it follows that H = h = 1| 2 1 2 TÿJ. Hence, we can rewrite:
Enio (
Enig
T
min
Enig
Tm
C H
Σ Eij T
C|
Enjj TH
C
= Σn
T
ij
= Σn 1TT
1k=
(25)
Th 2 0 by the definition of positive semidefinite matrix T and n. क
minimized, with a minimum value Ens; T Enis Stj. We therefore include the FLOPs budgets
discussed in Section 3.4 and Appendix B into our optimizations solve for non-trivial p.
D THE FEDDROP ALGORITHM
D.1 HIGH-LEVEL OVERVIEW
Algorithm Daaiimmmprrviiii provides an overview of the FedDrop FL algorithm. It accepts the client loss functions le with SyncDrop layers, client data weights Aes the FLOPS constraints function g [o, 1j + R, the SGD learning rate 77. the number of local epochs E the number of FL rounds R, and client sub-sampling ratio Φb. It returns optimized model parameters 0 (R+1) on the final round.
The algorithm starts by initializing model parameters (()) to be shared across all clients, and assigns uniform keep probability to p for all SyneDrop layers, which satisfies the FLOPs constraint ( (lines 2 and 3). After collecting trained model parameters ((r+1) from the current round r + 1 (line 6). the FedDrop server computes -0(-+1) i.e. the difference between the averaged model from the previous round ((r) with the trained model !!r+1) for client c (line 7, also in Figure 2
(r+1) for each neuron n between each pair of clients (i,j). It then measures the trajectory similarity 5tj as described on line and illustrated Figure Po. The term (r+01)+1) X,
computes the dot- dot-product between the weighted parameter updates of clients i and j. and max (ra)) pr)())) adjusts for the covariance from the synchronized dropouts of the previous round. Finally, the objective argming SS q) is optimized minimize the impact of model sparsity convergence.
D.2 COMPUTATIONAL OVERHEAD OF THE OPTIMIZATION OBJECTIVE
The complexity of computing the trajectory similarity in each round CW?)) where W the number of parameters and |C| is the number of participating clients. Additionally, the probability
19

---

Under review as a conference paper at ICLR 2022
Algorithm 1 The FedDrop algorithm.
1: function FEDAAGDROP( (Re, Ao): ce E,
2: random raudom_jnitializee(( (0 p)
b Initialize parameters and a uniform keep probability.
3: for 1,2.
4:
C + subsamplle (C, Ø))
> Sample a subset of clients.
5:
par for C
6:
0 o
f SGD, (L, ((0), Pcs E) A t Q 0
b Client training with SyncDrop layers. A Client update trajectories.
7:
8:
end par
for
9:
0
()r-1)
EeEc \ct(
Server aggregation with FedAvg- > Trajectory-similant estimation.
10:
for
i C, E C, E N do
1 + max p A0 -00
11:
12:
end for
> Optimal dropout probabilities. Objective from 7
13:
f argginng obj(S. q) s. ,(g) 20
14:
end for
15:
return ((+1))
16: end function
optimization stage on the server has complexity of C( where is the number of opti- mization iterations, |N] is the number of channels in the model. As an example, the VGG-9 model with 20 clients and I 10k steps of optimization would require only 53 FLOPs for computing the trajectory similarity, and < 448 G FL.PSs for the optimization steps, which can often exit early because it has converged. In practice, the server optimization only takes less than 1 minute on average. For reference, if we use local epochs per round E = 8, and FLOPSs ratio r = 0.5, the 20 clients require total of 25.8 T FLOPs per round. Overall, under this setting the additional computation overhead incurred by the server is only about 2% of total computational costs. For more practical consideration, it noteworthy that typically, the server is much less constrained in terms of computational power compared to the edge clients. In addition, given the above example, a single client requires 1.29 T FLOPs and the server requires 448 G FLOPs in total. The former consumes multiple hours of training in realistic implementation
iPhone 12 Pro (extrapolating from the profiling result in Table 10). whereas the latter only requires minutes to complete on an Nvidia V100 GPU.
D.3 AN EXAMPLE HISTORY OF PROBABILITY DISTRIBUTION
To better understand the probability optimization, we plot the histogram of optimized probabilities for each training round in Figure reveals that after multiple rounds of training, FedDrop gradually learns the importance of channels and start to make certain channels contribute less to the model training. Note that the histograms only provides distributions of probability values of all clients, each client may have vastly different optimized probabilities for the same channels.
D.4 MEMORY REDUCTION
As SyncDrop layers can sparsify models, many channels in the activation map can simply be skipped and never stored in both the forward and backward passes. For this reason, FedDrop can further reduce the memory cost associated with training. As an example, VGG-9 equipped with Syno can see up to 40% reduction in the memory requirement for training. We summarized the memory reduction for different scenarios in T
D.5 ACCURACY OF FLOPS ESTIMATION
To confirm the accuracy of our FLOPs estimation with the expected values, in Table sample the dropout layers in the Fashion-MNIST model with FLOPs budget ratios r e {0.25, 0.5, 0.75), can compute the actual FL.OPs counts. Under the setting of R 100 training rounds and local epochs
20

---

Under review as a conference paper at ICLR 2022
40
80
140
0.12 0.16 0.20 0.24 0:88 0.32 0.36 0.40 0.44 0.48 0.52 0.56 0.60 0.64 0.68 0.72 0.78 0.80 0.84 0.88
0.82
0.95 1.00
Figure 9: We present the evolution of dropout keep probabilities (horizontal axis) for multiple rounds of training (vertical axis) for VGG-9 training with FLOPS budget ratio 0.5 and the number of local epochs per round E is 4. After a few rounds of training, Fedprop can learn to distinguish channel importance.
Table 4: Theoretical memory reductions of FedDrop with varying batch sizes.
Batch size
Feature (MB)
Params (MB)
Total (MB)
FLOPs ratio
FedDrop Total (MB)
Reduction (%)
8.08
8.53
3.02
5.82
8.83
7.33
17.07
0.75
10.34
12.72
6.03
5.82
0.5
8.83
25,45
0.75
14.87
16.86
16
12.06
5.82
17.88
0.5
11.85
33.73
0.75
23.91
20.14 14
32
24.13
5.82
29.95
0.5
17.88
40.28
per round E = 8. We found the error to be within 0.06%. The low deviation of the expected value from the actual result is an effect of the law of large numbers.
Table Comparing the estimated and actual FLOPs counts of FedDrop. The figures within parenthe- ses represent the sampled FLOPs of 3 experiments with different random seeds.
FLOPs budget ratio r
0.25
0.5
0.75
Expected (T FLOPs)
140.97
281.85
422.50
Sampled FL.PPs)
(140.88,
140.91, 140.92)
(281.78,
281.81, 281.78)
(422.47, 422.48, 422.46)
Relative error (mean)
0.047%
0.021% %
0.0071 %
Relative error (std)
0.012%
0.005%
0.0019%
E EXPERIMENT CONFIGURATIONS
Models. Here, we provide the layout of the models used by the respective datasets. All models listed for computer vision (CV) tasks (in Tables ( feed- forward CNNs. Table the character-lewe single-layer LSTM model used for the natural language processing (NLP) task. Note that sparsity for the L.TT model were introduced for the hidden state and memory cell neurons.
21

---

Under review as a conference paper at ICLR 2022
Datasets. The experiments carried out three popular CV datasets (CIFAR-10 (Krizhevsky et al. 2014), Mushion-MNISS (Xiao et al. 2017) and SVHN (Netzer et al. 2011)) and natural language dataset (Shakespeare (Caldas et al. 2018b)). We the standard train/test split these datasets and report accuracy on test dataset with global model on server-side. To simulate different heterogeneity levels, for CV tasks split the datasets by class label with the distribution D/c (a) (Hsu et al. 2019 Wang et al. 2020b), and adjust the a constant for varying the degree of data heterogeneity. a U.5 unless specified for all experiments. Simple data augmentation was also employed for the training images, including random crop and normalization, and random flip in addition for CIFAR- 10. For the Shakespeare NLP dataset, follow LEAF (Caldas al. 2018b) for next-character prediction, and consider each speaking role in each play as individual client, and the resulting splits are thus inherently heterogeneous since each local client only contains the corresponding role's sentences.
Hyperparameters, We consider different hyperparameter settings for different tasks:
Client Sampling. For full client participation, we use 20 clients for CIFAR-10 and 100 clients for the remaining CV and NLP tasks. To simulate the fractional client participation, we randomly sample 10 out of 1000 clients on all datasets for CV tasks and 10 out of 660 clients on Shakespeare dataset for NLP task.
Optimization. As for CV tasks, searched the hyperparameters and found setting batch size B and learning rate = 0.02 were universally optimal for most methods. For NLP task Shakespeare dataset, the batch size B 32 and learning rate v 0.2 similarly found and made constant for all methods under comparison.
noticed that the reported accuracy could further improved if deeper models are used for baselines as they However, we are not pursuing state-of-the-an art accuracy on these datasets but instead comparing FedDrop with other methods under fair experimental settings and validating its effectiveness. We also noticed that there exist larger alternative datasets (Hsu et al. 2020) for evaluation. Nevertheless, evaluating and reproducing all competing FL methods such large dataset infeasible given the limited (university) computational budgets. Alternatively, we aim to conduct a fair empirical evaluation that demonstrate the effectiveness of FedDrop under broad variety of scenarios.
exhibit greater redundancy and thus benefit the performance of FedDrop.
Table 6: Layout of the model used for Mishion-MNIST training.
Layer
Kernel
Stride
Feature shape
#Params
#FLOPs
Conv4ReLU
5
32
28 x 28
832
652 k
2
Max Pool
2
32
14
25.1
3
Conv+ReLU
5
64
14
51.3
10.0 M
4
Max Pool
2 2x
64 x 7
125 k
5
Conv+ReLU
3 3x3 3
64 x
36.9 k
923 k
6
Avg Pool
x
64 x x
1.6
7
FC
512
132
132
8
FC
10
5.13
5.13
Total
226
11.8 M
Table Layout of the model used for SVHN training.
Layer
Kernel
Stride
Feature shape
#Params
#FLOPs
Conv+ReLU
5
32
28
28
2.43
1.91 M
Max Pool
2
32
14
25.1
3
Conv+ReLU
5
64
10
51.3
5.13 M
4
Max Pool
2 2
64 x
5
x
6.4 k
5
Conv+ReLU
3 3x3 3
64
x
x
x
36.9
333 k
6
Avg Pool
2x2
64 x
x
576
FC
512
132
132
8
FC
10
5.13 k
5.13 k
Total
227
7.54 M
22

---

Under review as a conference paper at ICLR 2022
Table 8: Layout the VGG-9 model used for CIFAR-I0 training.
Layer
Kernel
Stride
Feature shape
#Params
#FLOPs
1
Conv+ReLU x 3
32 x 32 x 32
896
918
2
Conv+ReLU
x
I
64 x 32 32
18.5
18.9 M
Pool
2
64 X x
65.5
4
Conv-ReLL
x
1
128 x 16 x 16
73.9
18.9 M
5
Conv+ReLL
3 x
I
128 x 16 16
148
37.8 M
6 Max Pool
2 x
2
128 8 8
32.8
Conv+ReLU
3
295
18,9 M
8
Conv+ReLU
3 xx3
1
256 8
590
37.8 M
9
Avg Pool
8
256 x x1
16.4
10
FC
512
132
132
11
FC
512
263
263
12
FC
5.13
5.13
Total
1.53 M
134 M
Table 9: Layout of the model used for Shakespeare dataset training.
Layer
Shape
#Params
#FLOPs
Encoder 80 x
640
LSTM
x 256
272 k
275 k
3
Decoder
256 x 80
21 k
21 k
Total
293 k
296 k
ADDITIONAL EXPERIMENTAL. DETAILS AND RESULTS
F.1 TRADE-OFF CURVES
Figures 10 to 12 find the optimal configurations for each competing FL methods that produces their corresponding communication/computttiio trade-off curves. is notable that FedDrop can extend the trade-off relationships much further and shows Pareto dominance against other algorithms under most conditions. these figures, the configurations explored include combinations of FLOPs budget ratios E {0.25, 0.333, 0.5, 0.75) and local epochs per round E' E {1, 4,8, 16).
10a
10a
Faltinp
Feltipp
Communication (GB)
Communications (GBB
FaddAg
SCAFTOLD
SCAFFOLD
10
104
1014.2 1014.4 1014. 1014. 1033 1013.2
1044.6
1014.8
10th
1015.2
1015.4
Number of FLOPs
Number of FLOPs
(a) 70%
(b) 80%
Figure 10: The FLOPs vs. communication trade-off curves of different FL methods reaching target accuracy for CIFAR- 10. We highlight the Pareto optimal points with dotted lines.
23

---

Under review as a conference paper at ICLR 2022
Redibp
10a
Conmunication (GB)
Utilop
Communication (GB)
Unitop
Follng
Falley
PelPhis
104
SCAFFOLD
SCAFROLD
104
10a
10a
10
10
10
10
10'3.
1013.
1013
10
10'4
Number of FLOPs
Number of FLOPs
(a)
80%
(b) 85%
Figure 11: The FLOPs vs. communication trade-off curves for BashionMMNIST.
Fealhop
FedDoup
Communication (GB)
Unithop
Communication (GB)
Unihepp
Pedlog
104
00
SCAFFOED
104
SCAFFOLD
10a
10a
1012.
1013
1013.2
1023.4
1013.6
1043
1023.2
1013.4
1013.
1013.
Number of FLOPs
Number of FLOPs
(a) 80%
(b) 85%
Figure The FLOPSs vs. communication trade-off curves for SVHN.
F.2 DATA IMBALANCE
examine the effect of data imbalance on the performance of FedDrop. Again, it shows that Fedrrop can manage the fastest convergence when compared other methods, and an increase in data imbalance further enhances its effectiveness against the others.
Figures 13 to 15
80
60
Accurncy (%)
Accuracy (%6
60
60
Accuracy
Rellee
400
40
Dilbey
Falleg
RedArg
Faddes
20
20
20
cla
Cakkon
0.2 0.4 0.6 0.8
1.2
0.2 0.4 0.6 0.8
1.2
0
0.2 0.4 0.6 0.8 1.2
Number -1015
Numher FLOPPs -1035
Number FLOPs -1015
(a) E 1, 5.
(b) E 1, 0.5.
(c) E 1, 0.05.
Figure 13: Varying a for the dataset splits (a) of CIFAR-
F.3 VARYING LOCAL EPOCHS PER ROUND
Figures I6 and 17 adjust the number local epochs per round, and observe that with more frequent rounds (lower P.). the accuracy gaps between FedDrop and the other methods become greater.
24

---

Under review as a conference paper at ICLR 2022
80
60
Accuracy (%)
Accuracy
60
Fallbeg
Accurscy
60
.00
Rebbee
Usilor
Todens
40
400
Ruttis SCAFFOLD
SCAFFOLD
SCAFFOLD
20
20
10th
10
10
10
10
10
Number FLOPs
Number of FLOPs
Number FLOPs
a
5.
(b) 4, a 0.5.
(c)
4,
0.05.
80
80
Accumcy (%)
Accuracy
60
Accumcy
60
PedDire
800
TeDlop
Dupp
Dxilaey
Falleg
Fellyg
40
Falboo
40
Pedlros
40
Padtos
SCAFOOD
SCAPPOLD
SCAFROLD
20
20
20
101a
1014
10ta
1024
10
1014
Number FLOPs
Number FLOPs
Number of FLOPs
(d) = 5.
(c) E 8, 0.3.
(f) 8, a 0.05.
Figure 14: Varying for the dataset splits D ca(( (a) of Fashion-MNIST, scale is used for clarity.
80
80
Accuracy (%)
Accuracy (%)
60
Accuracy
60
RedDrcy Unibee
60
Fallop
UsDtop
Felling
40
40
40
SCAFTOLD
SCAPPOLD
SCAFFOLD
Calboceal
20
20
20
10a3
10
1013
10a4
10
10
Number of FLOPS
Number FLOPS
Number of FL.Ops
(a) F 1, a 5.
(b) E 4, 0.5.
(c) E 4,
0.05.
80
80
Accuracy (%)
60
Accurcyy
60
Pedtbey
Accuracy
260
UnDoy
Uxibbry Pebloz
UnClop
Fodlog
40
40
SCAFFOLD
SCATTOLD
Calticeeal
Caltic
20
20
20
1043
1614
10t3
1044
1018
10-4
Number of FLOPs
Number of FLOPs
Number of FLOPs
(d) E 8, = 5.
(e) E = 8, 0.5.
(I) E 8, 0.05.
Figure 15: Varying a for the dataset splits (ac)(a)) of SVHN. Log-scale used for elarity.
F.4 VARYING FLOPs BUDGET RATIO
Figures 18 to 20 explore varying FLOPs budget ratios r E {0.75, O.5, 0.333, 0.25]. We found that
higher sparsity (lower ratio) can improve the overall convergence rate w.E.. FLOPs expended. However, the final converged accuracy may degrade; this can be fixed by simply adjusting the budget ratio approach 1.
25

---

Under review as a conference paper at ICLR 2022
90
90
85
85
855
Accuracy
FeDDp
Accuracy
FaDhup
Accuracy
Fellbay
80
80
80
Ukilrez
Toding
Pedlvg
75
TecPon
FadPrnn
75
FedPos
70
0.2
0.4
0.6
0.8
2
3
5
0.5
1.5
2
2.5
Number FLOPs -1044
Number of FLOPs
-1013
Number FLOPs
-1033
(a) E 4.
(b) E 2.
(c) B = 1.
Figure 16: Comparing FL. methods with the same local training epochs per round E used for SVHN.
90
85
Accuracy (%)
Accurucy (56)
80
Accunccy
80
Pedbee
Labeey
80
UsDDpp
PeDDhop
Polthep
Uuiboy
Felhig
75
75
Kdhiss
75
Pellhoo
Pethas
SCASTILD
SCATFOLD
SCAPTOLL
70
70
70
0.2 0.4 0.6 0.8
1.2 1.4
2
3
5
0.5
1.5
2
2.5
3 3.5
Number of FLOPs
"1
Number of FLOPs -10A5
Number of FLOPs -10
(a) = 4.
(b) E = 2.
(c) E == 1.
Figure 17: Comparing FL methods with the same local training epochs per round E used for Fashion-MNIST.
F.5 FRACTIONAL. DEVICE PARTICIPATION
We increase the number of all devices to 1000, and reduced the fraction of participating devices per round to 1% for each of the datasets. All competing methods perform uniform device subsampling each round as this was commonly reported for their original experiments McMahan et al. (2017); Li al. (2020): Karimireddy et al. (2020); Caldas et al. (2018a). For FedDrop to perform well, we uniformly sample devices every 2 rounds, to train with freshly updated probabilities. In Figure 21 Figure 22 Figure 23 and Figure 24 we compare FedDrop against all competing methods, and show that it can bring notable improvements in both communication and computation efficiencies.
F.6 ACTUAL TRAINING RUN TIME
In Table 10 we provide the average training times per image on an iPhone 12 Pro running the Fashion- MNIST model with a batch size of 4. Here, each result obtained under optimal conditions where no frequency throttling occurred. The structure of the model can be found in Table Appendix E Note that
only able report CPU run times, Core MI. (cor) 2021) currently only support training on CPUs. We implemented custom convolutional layer to take advantage of the sparse input and output feature maps.
Table 10: iPhone 12 Pro run time of training CPUs.
FLOPs ratio
Theoretical speed-up
Average run time image (μs)
Actual speed-up
Average (HS)
5 runs
1.0
1x
966.64
867.34
832.46
1072.06
1178.76
983.45 + 128.50
1.00x
0.5
2x
544.40
551.54
567.80
568.44
576.32
561.70 + 11.83
1.75x
0.25
4x
388.04
372.54
378.06
380.22
379.40
379.65 + 4.98
2.59x
26

---

Under review as a conference paper at ICLR 2022
80
80
80
Accuracy (%)
Accuracy
Accuracy
40
40
40
05
20
20
10
10
1.44.
10
10
104
10
10
10
10
10
Number FLOPs
Number FLOPs
Number of FLOPs
(a)
(b)
(c) E 8.
80
80
Accuncy (%)
Accurncy (%)
60
60
Accuncy
40
Tadlay
40
Fedihig
40
20
20
0,33
20
2.333
0
20
40
60
80
20
40
60
80
0
20
40
60
80
Number of Communication Rounds
Number of Communication Rounds
Number of Communication Rounds
(d) E = 2.
(c) E = 4.
(f) E = 8.
Figure 18: Reducing the FLOPs budget ratio r for CIFAR-10.
80
80
80
Accurcyy (%)
60
60
Accuracy
Accuracy
40
40
40
D.5
0L5
0335
20
20
20
10
10
10a2
104
12.2.
10ah
10.
Number FLOPs
Number of FLOPs
Number of FLOPs
(a)
(b)
(c) E 4.
80
80
Accuracy (%)
Accuracy (%)
60
Accurey
80
60
40
05
40
0,355
033
0.333
20
20
20
10
20
30
10
20
40
20
30
40
Number of Communication Rounds
Number of Communication Rounds
Number of Communication Rounds
(d) F = 1.
(c) F = 2.
(f) F =.
Figure 19: Reducing the FL.OPs budget ratio r for Fashion-MNISS
For the FedAvg motivation example in Section [ we assumed 20 clients, with 2500 examples per epoch each client, and 4 training epochs per round, which is in line with experiments with 20 devices. We extrapolate the results to 100 epochs for a batch size of 4 to estimate the run time to be 983.45 jss x 2500 4 x 100 16.39 min without frequency throttling. As each parameter is stored as bit float, each client would upload and download a model of size 226 & x 4B 904 kB once per round; the overall transmission per client is thus evaluated be 904 AB 2 100 176.56 MB.
27

---

Under review as a conference paper at ICLR 2022
80
80
80
Accuracy (%)
60
Accuracy
40
40
40
20
20
10 10t2 1
10
10
26 100 10 2. 10 10
10
10 10t26 10 10
10
Number FLOPs
Number FLOPs
Number FLOPs
(a)
(b)
(c) E 4.
80
80
Accunccy (%)
Accuracy
80
Accumcy
60
60
40
40
0253
0.33
0,33
20
20
20
10
15
20
25
30
35
40
10
20
30
40
10
20
30
40
Number of Communication Rounds
Number of Communication Rounds
Number of Communication Rounds
(d) E
1.
(c) E = 2.
(1) E = 4.
Figure 20: Reducing the FLOPs budget ratio r for SVHN.
10
relliep beillpp
restieg
Communicaiion (GB)
Communisations (GB)
Dalloy
105.5
zathee Cataicral
Nilha Callacr
10a
10
10
10
10
10
10
10
10:48
Number of FLOPss
Number of FLOPs
(a) 70% for CIFAR-10.
(b) 75% for CIFAR-10.
Figure 21: Comparing the FLOPs vs. communication trade-off across different FL methods reaching a target accuracy for CIFAR-10. We highlight the Pareto optimal points with dotted lines.
F.7 ADDITIONAL RESULTS ON THE MOTIVATING EXAMPLE
For reference, in Figure 25 we included all parameter update magnitudes of the first convolutional layer for the motivating example given in Figure in Section
F.8 RESULTS ON THE NLP TASK
We provide results on Shakespeare dataset for full client participation in Figure 26 Figure 28 and fractional client participation in Figure 29 Figure 31 The experiment configurations are introduced detail in Appendix
28

---

Under review as a conference paper at ICLR 2022
10
Futhhp Uixtip Fitlyg
PoLong
Communication (GB)
Communications (GB)
10
10
10a
10
10
10
10126
10
10
10
Number FLOPs
Number FLOPs
(a) 80% for Fashion-MNNT
(b) 85% for Fashion-MNiSt
Figure 22: Comparing the FLOPs vs. communication trade-off across different FL methods reaching target accuracy for MNNIS. We highlight the Pareto optimal points with dotted lines.
restlive Lhathop Aidlie
109
Communication (GB)
Communications (GBB)
Follig
10
10a
10a
10 10
10
10
10
1022
10
10:24
Number of FLOPs
Number FLOPs
(a) 80% for SVHN.
(b) 85% for SVHN.
Figure 23: Comparing the FLOPs vs. communication trade-off across different FL methods reaching target accuracy for SVHN. We highlight the Pareto optimal points with dotted lines.
80
80
80
Accuncy 60
Accuracy (36)
Accuracy
lettoo Billop
60
60
40
SCAPHOLI
SLANCLD
20
20
10ah
10.2
1043
Number of FLOPs -104
Number of FLOPs
Number of FLOPs
(a) CIFAR-10 E =
(b) Fashion-MNIST with
(c) SVHN with
Figure 24: Reducing device participation to 1%. Some plots use log-scale for clarity.
29

---

Under review as a conference paper at ICLR 2022
+
Figure 25: All parameter update magnitudes for each channel (row) in the first convolutional layer for all clients (column) after I round of training. The pairs of clients with same colors (one lighter and one darker) signify that they shared the same image class. The length of each ray is the magnitude of the parameter update, and the parameters are ordered by the angles of the rays. The results are taken from the motivating example in Figure I
30

---

Under review as a conference paper at ICLR 2022
FedDrop
Communications (GB)
UniDop
FodArg FodProx
104.5
Caldes
104
10
1015.2
10
Number of FLOPS
Figure 26: Comparing the FLOPs vs. communication trade-off across different FL methods reaching a 50% accuracy for the Shakespeare dataset. We highlight the Pareto optimal points with dotted lines.
50
50
50
Accuracy (56
Accuracy
40
Accuracy
40
Festinp Exibeey
TaDlv
30
30
30
Bullop
bethes
SCATTEED
20
SCAFFOUT
SEATROLD
Calbreral
10
10
10
4
Number of FLOPs -10
Number of FLOPs -10
Number of FLOPs -10
(a) 0 0.5, E = 1, 0.5.
(b) a 0.5, 2, = 0.5.
(c) a = 0.5, E 4, 0.5.
Figure 27: Comparing FL methods with the same local training epochs per round E used for Shakespeare.
60
60
50
50
50
Accurncy
40
Fedtbey Usibag
Accuracy
Foitbop Uniloo
Accuracy
40
EedDbop
30
30
30
redfies
SCAFFOLL
SCAFFOLD
20
SCAFFOOD
10
10
10
0
100
20
80
20
60
100
Number of Communication Rounds
Number of Communication Rounds
Number of Communication Rounds
(a) 1, r = 0.5.
(b) F = 2, 0.5.
(c) F = 4, = 0.5.
Figure 28: Comparing FL methods on the number of communication rounds ys. accuracy with the same local training epochs on Shakespeare.
31

---

Under review as a conference paper at ICLR 2022
FedDoop
Communications (GB)
UniDrop
FodArg
103.5
FedProx
Caldis
103
10a4 10 10 1021.6 1024.8 Number of FLOPs
Figure 29: Comparing the FLOPs vs. communication trade-off across different FL methods reaching a target accuracy for a given dataset. We highlight the Pareto optimal points with dotted lines.
50
50
50
Accuracy (%6)
Accuracy (56
40
Todhev
Accuracy
Festinp Eribeey
TaDlve
30
UsDbpp
30
30
RCATFOED
20
SCAFFROU
SEATROUD
Calticiral
Cilla
10
10
10
0.2 0.4 0.6
0.8
0.2 0.4
0.S
0.5
2
Number of FLOPs
-10*
Number of FLOPs -10
Number of FLOPs -10"
(a) a 0.5, E' = 1, 0.5.
(b) a 0.5, 2, = 0.5.
(c) a = 0.5, E' 4, 0.5.
Figure 30: Comparing FL methods with the same local training epochs per round E used for Shakespeare.
60
50
50
50
Accuracy (%h
Accuracy
400
10
Fefflen Eeiliigg
Accuracy
30
Us Diep
30
30
Todhho
20
SCAFFOOD
SCAFROLD
SCAFROLD
10
10
10
20
60
100
40 Number of Communication Rounds
100
20
80
Number of Communication Rounds
Number of Communication Rounds
(a) F = 1, = 0.5
(b) = 2, r = 0.5
(c) F. = 4, 0.5
Figure 31: Comparing FL methods on the number of communication rounds ys. accuracy with the same local training epochs on Shakespeare.
32


Reviews:
[{"review_id": "wl3W3ZCxp1i", "reviewer": "Reviewer_A6bP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The paper tackles the problem of computational cost to the clients, which is relatively understudied compared to the communication costs. \n- The technique does not make any simplifying assumptions and is applicable to real world FL use cases. \n\nWeaknesses\n- The method proposed comes with increased communication costs to the client. While the authors show that they can favourably trade communication costs for computational costs, generally practitioners are more concerned with communication costs, as these can impose real costs on the users (such as using up limited wifi resources), whereas computational costs are largely in training time which are 'paid' by the server. \n- The results here are very specific to computer vision. Would similar results be possible for other problems such as NLP?\n- The log graphs in figure 5 are odd and challenging to read, with different spacing applied to the x and y axis. \n- In figure 5.c how is FedDrop able to achieve lower communication costs than FedAvg?\n- The results in table 1 are a little difficult to interpret. If I understand them correctly, they indicate the computational cost of different method to reach a given accuracy, under the constraint that they keep below a certain communication budget. If this is correct, it would be much more valuable to the reader to record the actual communication costs of each method as well as their computational costs. Simply saying that all methods stayed below some budget may be hiding important information about the additional communication costs of FedDrop. \n- Empirical results appear quite weak, with a 10x - 100x increase in the communication costs resulting in less than a 10x reduction in computation costs. ", "summary_of_the_paper": "The authors propose a new method of coordinated, per client and weight dropout for federated learning. The intuition behind the method is to increase the dropout probability (probability of having a weight set to zero) for weights where different clients often have opposite parity gradients (as when those gradients are summed together are likely to cancel each other out). The authors show that doing this can reduce the computational cost on clients of FL by up to 3x.  ", "summary_of_the_review": "While reducing client computational costs is an important task in FL, the method proposed in the paper comes at a very high communication costs, which makes it unlikely to be practically useful while communication costs are still the primary concern in FL. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_s2nlXfNSJ", "reviewer": "Reviewer_wCnx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The premise of the paper that other works actually over-emphasize communication, while local computation is as time-consuming is a bit misleading. I agree with the fact that the act of communicating information, i.e. sending the updated model parameters over the channel might not take so long. But this is not the only factor defining the time that each communication round takes in FL. There is a lot of overhead associated with each FL round: 1) select an available device (usually connected to wifi and power) 2) make sure the device does not become unavailable,... 3) collect enough updates from enough devices each with their own delays (depending on the loads, computation power...); note that here we are constrained by the slowest device. All these factors actually add a lot of overhead to each FL round. So just looking at the communication piece without seeing this bigger picture is misleading. In fact, all the methods that are compared against are targeting the reduction of the number of rounds and not communicated bits; if communicated bits were the target, the authors should have compared against methods that directly target that, e.g. by compressing the updates. \n- The theoretical results that are presented by the paper are very weak. Their statements are misleading/sloppy or wrong and lack assumptions. And to be honest the theorems should not be called that.\n  - Theorem 1: This requires Assumptions 1-2 which are very strong and definitely not true for all the layers. And the result under these assumptions is trivial.\n  - Theorem 3 is just obvious!\n  - In Lemma 1, there is no reasonable justification to drop the Hessian term. It cannot be justified and all the approximations based on that are not valid. Note that Lemma 2 and Theorem 2 are built on top of the closeness of this approximation! Finally, although the results are presented for gradients, there is a big leap when using this in practice, where the updates are actually the accumulation of multiple steps instead of gradients.\n- The algorithm is impractical and possibly not very useful in many cases:\n  - No secure aggregation can be integrated into it, while many FL systems require secure aggregation.\n  - The steps on adjusting the drop-out probabilities are computation/memory/communication intensive. First of all the computation there scales with the number_parameters x batch_size^2. which with large batch sizes in practice could be too much. More importantly, it will require the server to track p's for all the clients which is not possible in large-scale applications. The other alternative is that the clients keep sending the server their p's which will increase the communication as the size of p's is large. In fact, I suspect for large-scale realistic problems with many clients and a low percentage of participation per round (which are mostly absent from the experiments) this stateful situation would not necessarily lead to any benefit (as the clients' p's are updated so infrequently). I believe that is why in the case where there are 1000 clients and 1% participation the authors have had to resort to choosing the same clients for two consecutive updates; see appendix F.5. One possible issue with this is that it is not possible to make sure that the same set of clients are available for two consecutive updates due to external situations (clients may become temporarily unavailable at any point during the training). \n- The experimental results are not convincing and I do not think the gains are significant enough.\n  - The method is only tailored and only tested on CNNs while many FL problems do not use them. This has made the experimental section less convincing as there are many usual FL benchmarks that are absent.\n  - Flops are not easy to estimate and most of the reported results are in terms of FLOPS. While even the authors' estimates of actual FLOPS savings show that the theoretical numbers used for FLOPS (as the x-axis in many figures) could be as much as 1.5x worse; see table 9.\n  - There are only a few plots comparing the rounds, while that is important. In fact, there is actually only one plot (Fig 18) that compares FedDrop with another method FedAvg over the number of rounds. And in that plot, it is clear that FedDrop cannot achieve the final accuracy of FedAvg with low r (which is the setting that has gains based on other plots). And in order to achieve similar final performances to FedAvg, FedDrop has to run with r=0.75 or higher which does not seem to result in any significant gain (in other metrics).\n  - I noticed that the results that show a significant gain for FedDrop, e.g. Fig 4, actually use very low E; compare Fig 4 to Fig 8, where E=8. Note that large E is essential in getting good results in FL. Also, this is very unfair to other methods as their only tool in reducing the communication rounds is local computation; while FedDrop is additionally using r! Based on this I do not believe the authors have done a fair comparison. It is worth noting that in many of the plots it seems that running the other algorithms for a longer time could also result in better results than FedDrop (e.g. see Fig 15); Interestingly, even in this case Fig 16 shows a large gap in favor of FedDrop which is due to the use of low E as well! Unfortunately, I do not think the authors use good criteria to stop the algorithms (e.g. when they have converged).\n\n======= After Rebuttal =======\n\nI do not think the authors' response addressed my concerns.\n- Around the theoretical results: my original comment was around the fact that just stating some theoretical results with assumptions that are not realistic or verified does not add to the paper's value. Moreover, the results in Theorem 3 are obvious. And the term Theorem is reserved for significant contributions and does not apply here. Also, the authors did not provide any meaningful justification for the approximation in Lemma 1 (and the divergence from theory where there are multiple local steps).\n- For the theoretical results: As other reviewers have also mentioned, the empirical results are quite weak. And as I mentioned in my original review they are not fair (not running the algorithms for long enough, not allowing larger E for other methods, ...). Unfortunately, the authors did not properly address these issues in their rebuttal. Please refer to my original review for detailed issues regarding the empirical results in the paper.", "summary_of_the_paper": "The paper proposes FedDrop and a synchronized drop-out strategy to reduce local computation during the training of FL models.", "summary_of_the_review": "Based on the above comments, I do not believe this paper passes the acceptance threshold. I encourage the authors to address the points around: motivation, practicality, and rigorous experimental comparisons with other works.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RjxgeElGxkK", "reviewer": "Reviewer_guLZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The papper consists of following strengths and weaknesses\n\nStrengths:\n- The dropout mechanism is simple yet effective in deep learning. It is interesting to see how to use dropout to design new federated learning algorithm.\n- Extensive numerical experiments verify the advantage of FedDrop.\n\nWeaknesses:\n- The paper lacks discussion on the convergence of the algorithm.\n- The paper only shows experiments with neural network examples. I wonder how the methods perform with models that are not neural networks.\n- The algorithm requires to solve an extra sub-optimization problem. The cost of solving this problem has not been well-discussed in the paper. In FL, with large number of clients, i.e. the dimension of q also grows. I am concerned about the efficiency of solving this subproblem while other baselines such as FedAvg/FedProx do not need to have this additional step.\n- Keeping the algorithm in the appendix reduces the clarity of the paper.\n\nQuestions:\n- As FedAvg does not work well with heterogeneous data, I wonder whether FedDrop also suffers in such setting as the update of FedDrop is very similar to FedAvg.\n- Also, do we expect the same convergence guarantee as FedAvg?\n\nMinor comment:\n- There should be square in equation (6) as it is MSE not RMSE.\n- In theorem 2, there should be description for the dot product, typo at \\Delta \\theta^n_j", "summary_of_the_paper": "The paper proposes a new method where local workers will drop part of their model using a shared dropout probability received from the server at each communication round. The dropout probabilities are computed by solving an optimization to promote similarity across agent's update. Compared with popular baselines in federated learning, the proposed method demonstrates better performance in terms of number of FLOPs.", "summary_of_the_review": "I believe the paper contain notable contribution in designing new algorithm using dropout. Numerical experiments illustrate the practical performance of the proposed algorithm However, the proposed lacks discussion on the convergence of the algorithm while other methods like FedAvg/FedProx do have guarantee. The proposed methods appear to only work directly with neural network models, there should be more discussion on other types of model.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NO_p6PsOBXl", "reviewer": "Reviewer_tPTk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. This similarity based dropout technique to reduce communication cost in FL is new, and the authors proposed novel algorithm to approach it.\n2. They also did sufficient experiments to empirically validate their method.\n\nCons:\n1. I think the writing and organization can be improved since I am not very clear about some technical details, which I will talk about below.\n\nQuestions:\n\n1. I am confused about  $\\boldsymbol{p}$ and  $\\boldsymbol{p}_c$. I was wondering for $\\boldsymbol{p}$, do we optimize a single $\\boldsymbol{p}$ or for each client c, we compute a  $\\boldsymbol{p}_c$? \n\n2. It seems that the constraint $g(\\boldsymbol{p})$ is a combinatorial function since it contains the number of channels. How do you optimize on it?\n\n3. For the experiments, I expect to see FedDrop can converge faster than FedAvg or SCCAFOLD in terms of the wall clock time since the main advantage of it is to reduce model size and hence speedup training.", "summary_of_the_paper": "This paper proposed a dropout based method towards communication efficient federated learning, where each client will compute the dropout probability based on the update similarity between local models. The experiments show that with fewer FLOPs than traditional federated learning algorithm, FedDrop can achieve the same accuracy.", "summary_of_the_review": "Overall, the idea of this paper is new, but I think this paper can be better organized to make reader easier to understand the technical details. If the authors can address my questions, I am willing to change my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces drop-out probabilities which are adaptive to the similarity of model parameters between clients.
The reviewers liked the idea, however missed several aspects, such as a convergence analysis or at least discussion, as well as an analysis of additional cost of the adaptive step, and finally several concerns on the strength of the experimental setup and benchmarks.

Unfortunately consensus among the reviewers is that it remains below the bar even after the discussion phase.

We hope the detailed feedback helps to strengthen the paper for a future occasion.
Decision: Reject