Revisiting Sparsity Hunting in Federated Learning: Why does Sparsity Consensus Matter?
Authors: Sara Babakniya
Souvik Kundu
Saurav Prakash
Yue Niu
Salman Avestimehr
Venue: Accepted by TMLR ()

Abstract: Edge devices can benefit remarkably from federated learning due to their distributed nature; however, their limited resource and computing power poses limitations in deployment. A possible solution to this problem is to utilize off-the-shelf sparse learning algorithms at the clients to meet their resource budget. However, such naive deployment in the clients causes significant accuracy degradation, especially for highly resource-constrained clients. In particular, our investigations reveal that the lack of consensus in the sparsity masks among the clients may potentially slow down the convergence of the global model and cause a substantial accuracy drop.
With these observations, we present \textit{federated lottery aware sparsity hunting} (FLASH), a unified sparse learning framework for training a sparse sub-model that maintains the performance under ultra-low parameter density while yielding proportional communication benefits. Moreover, given that different clients may have different resource budgets, we present \textit{hetero-FLASH} where clients can take different density budgets based on their device resource limitations instead of supporting only one target parameter density. Experimental analysis on diverse models and datasets shows the superiority of FLASH in closing the gap with an unpruned baseline while yielding up to $\mathord{\sim}10.1\%$ improved accuracy with $\mathord{\sim}10.26\times$ fewer communication, compared to existing alternatives, at similar hyperparameter settings.

Published in Transactions on Machine Learning Research (09/2023)
Revisiting Sparsity Hunting in Federated Learning: Why does Sparsity Consensus Matter?
Sara Babakniya
babaknig@husc.edu
Department of Computer Science University of Southern Cakfornía, USA
Souvik Kundu
souvikk kon.uukkknddddnttee.... com
Intel Labs, San Diego USA
Saurav Prakash Department of Electrical and Computer Engineering
suwreepr@usc.com edu
University of Southern California, USA
Yue Niu
yueriuuusc. edu
Department of Electricul and Compuler Engineering University of Southern California, USA Salman Avestimehr
avestime
Department of Electrical and Computer Engineering
University of Southern California, USA
Reviewed on OpenReview: https: // openneviee. net/ forum? id= sHyhdpsnyi
Abstract
Edge devices can benefit remarkably from federated learning due to their distributed nature; however, their limited resource and computing power poses limitations in deployment. A possible solution to this problem to utilize off-the-shelf sparse learning algorithms the clients to meet their resource budget. However, such naive deployment in the clients causes significant accuracy degradation, especially for highly cosotrrc-constraaned clients. In par- ticular, our investigations reveal that the lack of consensus in the sparsity masks among the clients may potentially slow down the convergence of the global model and cause sub- stantial accuracy drop. With these observations, we present federated lottery aware sporsity humling (FLASH), a unified sparse learning framework for training a sparse sub-model that maintains the performance under ultra-low parameter density while yielding proportional communication benefits. Moreover, given that different clients may have different resource budgets, present hetero-FLASH. where clients take different density budgets based on their device resource limitations instead of supporting only one target parameter density. Experimental analysis on diverse models and datasets shows the superiority of FLASH in closing the gap with an unprumed baseline while yielding to 10011% improved accuracy with 10.26x fewer communication, compared to existing alternatives, at similar hyperpa- ffl.fffllgg_
rameter scttings. Code is available https::/gttth:ii http://www...tgg.ooooooo///aaaaa
Introduction
Federated learning (FL) (McMahan et al. 2017 is popular form of distributed training, which has gained
significant traction due to its ability to allow multiple clients to learn common global model without sharing
their private data. However, clients' heterogeneity and resource limitations pose significant challenges for
FL deployment over edge nodes, including mobile and loT devices. To resolve these issues, various methods
have been proposed over the past few years for efficient hetorogeneous training (Zhu et al.] 2021 Horvath
et al. 2021 Diao et al. 2020 or aggregation with faster convergence and reduced communication Li al.
Authors have equal contribution.

---

Published in Transactions on Machine Learning Research (09/2023)
2020b; Reddi et al., 2020p. However, these methods do not necessarily address the growing concerns of high computation and communication limited edge.
Meanwhile, efficient edge deployment via reducing the memory, compute, and latency costs for deep neural networks in centralized training has also become an active area of research. In particular, recently proposed sparse learning strategies (Evci et al. 2020) Kundu et al. 2021 2022: Mocanu et al., 2018 Dettmers & Zettlemnoyer 2019) effectively train weights and associated binary sparse masks such that only a fraction of model parameters (density d << 1) are non-zero during training. This enables the potential for reduction in
both the training time and compute cost (Qiu et al. 2021 Raihan & Aamodt 2020 while yielding accuracy
close to that of the unprumed baseline.
In addition to the aforermentioned benefits of sparse learning in centralized settings, its proper deployment in FL can reduce communication costs. In particular, users can train larger models while communicating only a fraction of weights that are non-zero (Fig. 1 (a)).
However, the challenges and opportunities of spanse learning in FL are yet to be fully unveiled. Only
01.5
Your Accssccy 7N0
very recently, few works (Bibikar et al. 2022 Huang
8 8 88
22.5
et al. 2022] have tried to leverage non -ag-aggrsssve
22.5
model compression in FL. Another recent. work, ZeroFL (Qiu et al., 2021), has explored deploy- ing sparse learning in FL; however, failed to lever- age the advantages of model sparsity to reduce the clients' communication costs. Moreover, as shown in Fig, [bb,,, for d = 0.05, ZeroFL suffers from substantial accuracy drop of ~44% w.r.t baseline.
Gunseest, ewo.b
Gavgettlll Gesdeorm
Pockee owrt:
Communication 16
70.0
02
05
12
Figure 1: (a) Accuracy vs. Communication. For a given communication threshold, sparse learning FT, can improve performance. (b) Accuracy vs. parame- ter density in each client. Our approach significantly outperforms the existing alternative QQu et al. 2021
Contributions. Our contribution is fourfold. In view of the above, we first present crucial observa- tions in identifying several limitations integrating sparse learning in federated settings. In particular, we observe that the sparse aggregated model does not converge a unique sparsity pattern, primarily due to lack of consensus among the clients' masks in different rounds. In contrast, the model matures in centralized training, the mask also shows a higher convergence trend. We further empirically demonstrate the utility of incorporating layer importance and clients' consensus on the performance.
We then leverage our findings and present federated lottery aware sparsity hunting (FLASH), a methodology that can achieve compulation and communication efficiency FL by employing sparse learning.
Furthermore, in real-world scenarios, FL users are more likely to have highly heterogeneous resource budgets (Diao et al. 2020). Therefore, instcad of limiting everyone by the minimum available resource, extend our methodology to hetero-FLASHI where different clients can participate with different sparsity budgets yet yield sparse model, leveraging the resource and data of cach client while adhering to their own resource limit. Here, to deal with this problem, we propose server-side sub-sampling where the server creates multiple sub-masks of the global model's mask such that all follow the global layer importance.
We conduct experiments on MNIST, FEMNIST, CIFAR-10, CIFAR-100, and TinyImageNet with different models for both IID and non-III client data partitioning. Our results show that compared to the existing alternative QQuu et al. 2021), at isc-hyppeparrmeter settings, FL.SS can yield up to ~8.9% and 10.1%% on IID and non-lII data settings, respectively, with reduced communication of up to 10.2x
2 Related Works
Model pruning. In the past few years, a plethora of rescarch focused on model compression via pruning (Trankle & Carbin 2018 Liu et al.] 2021 You el al. 2019 Kundu & Sundaresan 2021 Niu et al.) 2020: Kundu et al. 2023a). Pruning essentially identifies and removes the unimportant weights to yield efficient inference models. To this aim, various methods have been proposed for different purposes. Here, we focus on Sparse Learning, which can potentially reduce the update size and compression errors (Xu et al.] 2020) for FL users.
2

---

Published in Transactions on Machine Learning Research (09/2023)
Sparse learning. It is popular form of model pruning that recently has gained significant traction Evei et al. 2020: Kundu et al. 2020 2019). In particular, in sparse learning with target density d, only d% of the model parameters remain non-zero during the training (d << 1.0 and sparsity is 1.0
In our paper, we leverage Dettmers & Zettlemoyer (019)) to find the sparse mask efficiently for each client. Particularly, the training starts with a sparse model and randomly initiated mask that meets the target parameter density d. Since there is no prior knowledge about the importance of weights and layers, we start with random sparse mask with uniform layer-wise parameter density d. The mask gets updated based prune-regrow policy at the end of each epoch. In each round, first, layers are ranked based on their normalized contribution in the non-zero weights. Then, the lowest I weights from each layer are pruned based on their absolute magnitude. Since this % pruning happens on top of d% densily, we need to regrow p.% from the pruned ones back in the model. Here, each layer gets the number of regrown weights proportional to their normalized contribution, giving layers with higher importance more non-zero weights. This process iteratively repeats each epoch to learn the mask and non-zero weights. Note that there are multiple choices for pruning and regrowing the weights; however, we chose absolute magnitude to save the cost of computing additional values, such as the momentum.
FL for resource and communication limited edge. Resource limitation and heterogencity are among the most known challenges, especially in cross-device federated learning (Kairouz et al. 2021). Existing works have explored the idea of heterogeneous training that allows clients to train on different fractions of full-model based on their compute budget (Horvath et al. 2021 Diao et al. 2020 Mei et al.
2022: Niu et al.,
2022). On a parallel track, various optimizations are proposed to accelerate the convergence, thus requiring
fewer communication rounds (Han et al., 2020 Gorbunov et al. 2021 Zhang et al. 2013 Li et al. 2019).
To make FL more communication efficient, few works have leveraged pruning (Li et al. 2020a Jiang et al.) 2022 Li et al. 2021). In LotteryFL (Li et al.) (020a), clients adapt personalized masks that perform well only on their local data. Morcover, the clients must regularly communicate the entire model to the server. Similarly, PruneFL (Jiang et al. 2022) also asks for high communcation costs as demands the participating clients to send the gradient values of the entire model to the server while updating the masks.
Only a few works (Huang et al.) 2022 Bibikar et al.) 2022 Qiu et al. 2021 tried to benefit from sparse learning in federated settings. In particular, Huang al. (0022) relied on randomly initialized sparse mask and recommended keeping it frozen during the training without providing any supporting intuition. FedDST (Bibikar et al. 2022), on the other hand, leveraged RigL (Evci et al.] (020) to perform sparse learning in clients. It relied on a large number of local epochs to avoid gradient noise and focused primarily on highly non-IID data distributions without targeting ultra-low density d. More importantly, neither of these works investigated the key differences between sparse learning in centralized and FL. With a similar philosophy as ours, ZeroFL QQiu et al., 2021) identified a fundamental aspect of sparse learning in FL; the top-k weights of clients are likely to differ for low densities; hence, the aggregated update cannot benefit anyone. They propose increasing the update density to increase the update overlap and improve accuracy. As a result, despite training a model with density d = 0.1, clients send updates to the server with d = 0.3. Thus, this approach does not benefit. from proportional communication elficiency; all clients receive a dense model and send back a 3x denser model, and still, the global model suffers from a significant accuracy drop.
Another contemporary work [Isik al.) 2023) leveragod the idca of learning the sparse mask while keeping the weights fixed to their initialized values (Ramanujan et al.; 2020). Thus, clients are only required to send the binary mask updates to the server, reducing the communication by 32x. However, due to SGD- based updating of floating point mask for cach weight, such methods do not necessarily help the client's 's local computation. Such an assumption is out of our current scope, as we assume an even stricter constraint where no client can perform dense model updates or storage while yielding significant communication savings.
3 Revisiting Sparse Learning: Why Does it Miss the Mark in FL?
In contralized training, applying sparse learning mothods has shown benefits in FLOPs reduction during forward operations (Evci et al., 2020), and potential training speed-up of up to 3.3x (Qiu et al. 2021 while
Prune rate (fr) indicates the of weights pruned from each layer of a sparse model.
3

---

Published in Transactions on Machine Learning Research (09/2023)
maintaining high accuracy at low densities (d S 0.1). However, here, the primary purpose of employing sparse learning is to utilize communication efficiency by reducing update size in FL. Such methods can potentially get better convergence and performance compared to post-training or data-independent methods.
Table 1: FL training settings considered in this work.
Datasot
Model
Data-
HoundsClient Clients / Round Optimizer A Aggrogation
Local
Batch
partitioning
(T)
(Cx)
(Ox,
spoch (E) size
MNIST
MNISTNet
400
CIFAR-10
600
LDA
100
10,
CIFAR-T00
NesNet18
800
SGD
FedAvg
TinthnageNet
PEMNIST Same
Caldas al. 2018
1000
3400
34,
In this section, we conduct an exhaustive analysis to understand how naive deployment of sparse learning in FL works while unveiling key differentiating factors in training dynamics with sparse lcarning in centralized compared to that of FL. We used a widely adopted FL training setting (Qiu et al., 2021 Diao et al., 2020 (refer to Table ! for details) on the CIFAR-10 dataset and added sparse learning on the clients. Here, each client separately performs sparse learning (Dettmers & Zettlemoyer 2019) to train a sparse Res.et118 model and moet a fixed parameter density d, starting from a random sparse mask. At the end of each round, clients send their updates to the server, where they get aggregated using FedAvg. We name this naive sporse training (NST) due to its plug-and-plas nature of deployment of sparse learning in FL. Notably, NST can decrease only communication from clients to the server mainly because these updates usually have different sparsity patterns. Therefore, aggregating them by simply averaging gradients causes the final model to have density >> d, translating to higher downlink communication cost.
BesNet18 CIFAB-10
ResNet18 CIFAR
Bes on CHAAN-1C (d-0.05 Cantralaed Reedwi18 on CIAR-10 (u c-005 Fedacated)
Test Accuracy
80-
37%
Test Accurscy
80-
25-
60-
60
325-
40
Centrallted Basetre
Centrakaed Baseiiee
34
Contralized
20
11
175
200
300 Round
500
200 Abc
500
2
Round
Spoch
Comm, Round
Figure 2: Accuracy vs. round. Comparison between the performance of sparse learning in federated and centralized settings (dense and sparse baselines are shown in dashed and solid lines.).
Figure 3: Heatmap of SM between masks generated in different rounds of (a) centralized and (b) feder- ated learning. Lower SMs indicate more similarilies between the masks.
Observation 1.
At ultra-low density d < 0.1, the collaboratively learned FL model significantly sacrifices performance, while the centralized sparse learning yields close to baseline performance.
Fig. 2 shows that sparse learning methods with high sparsity can perform similarly to the dense model (gray and green lines). However, naively deploying the same method in FL can cause significant performance degradation (purpic and orange lines). In particular, the same model can suffer from an accuracy drop of 3.7% and 12% for d = 0.1 and d = 0.05, respectively.
Observation 2.
As the training progresses, the sparse masks in centralized training tend to agree across epochs, showing its convergence, while the server mask in FL does lack consensus across rounds.
Definition 1. Sparse mask mismatch. Let us call the masks generated at rounds and j as M and M Now, we can define the sparse mask mismatch(SMI) smi) as the Jaccard distance between these two masks where Mi represents the sparse mask tensor for layer l.
( Mi
sm (1.))
(1)
MUMM
4

---

Published in Transactions on Machine Learning Research (09/2023)
Fig. 8 presents the SM for the intermediate sparse masks generated during sparse training in centralized and federated learning. As Fig-
3 a) shows, SM decreases in centralized learning as the training progresses. On the other hand, in Fig. 3 3(); the SM values for similar settings (same model, dataset, and density) for the global model in FL remains > 0.4 indicating a distinction in the sparse learning between the two settings.
Observation 3. In federated sparse learning with low target densily, a lack of consensus at the later (deeper) layer's masks remains more severe than that of the earlier ones.
Cone layer
DIFAR-10 Contralized!
Towardss mask agreement for layees
laynrs
jaccard Distancc
0.8
0--
0.4
Epoch
(b)
Comm. Round
105
Figure 4: Layer-wise sparse mask mismatch vs. epoch (rounds) for (a) centralized and (b) FL. In FL. the SM for later layers stays high, contrary to centralized, where SM reduces for all layers as the training matures.
Fig. 4 presents the layer-wise SM of the consecutive masks in centralized and federated learning. As shown in Fig. 4 [a), by increasing training rounds in centralized learning, for each layer, the distance (measured by SM) between masks decreases, showing a convergence trend. However, in FL (Fig. [b)), the later layers' masks change significantly in different rounds with SM value as high as ~..8. One possible explanation for this phenomenon is based on layer sensitivity, which we define below.
Layer sensitivity. All layers do not carry the same importance towards the final model performance. One proposed metric to measure this difference is layer pruning sensitivity (Ding et al.] 2019). After pruning the model using any prospective algorithms, layers that are assigned higher density are the ones that are more sensitive to pruning and play a more important part in the final performance. We can measure the sensitivity of a sparse layer (I) via the ratio its total # of non-zero weights to the total # of weights.
sensitivity' total# non-zero weights'
(2)
total # weights'
It is commonly known that later layers usually have lower sensitivity and a larger number of total parameters, allowing them to have many possible mask options compared to the earlier ones. Therefore, clients are unlikely to reach consensus and more likely to come up with different masks in the later layers, causing increased SM. For example, 90% of the parameters in layer I and 5% of the parameters in layer 14 are present in the final mask, and as expected, SM for these layers is 0 and ~0.73, respectively.
To further investigate the impact of high SM and layer sensitivity on a model's accuracy, we per- formed five different centralized training as described in Table 2 we randomly generate sparse masks with uniform density for all the layers. For rows 2 - 5, first, randomly create each layer's mask by following its pruning sensitivity?] then decide to freeze (prevent changing) the mask partially or for all layers. More specifically, in row 2, the initialized (pro-defined) mask does not change. For rows 3-5, al each epoch, a fraction of the mask in the specified layers changes to meet the target SM value. As Table shooss that large SM of masks between epochs can degrade the accuracy by up to 9.34%, and we can safely conclude that disagreement of masks across epochs can affect the final performance. Moreover, the model trained via sparse learning with sensitivity-driven pre-defined masks (row 2) yields better performance than the one trained using a pre-defined mask.
Particularly for the training of row 1,
Table 2: Performance based on the different levels of mask disagreement in centralized.
Training
Use sena- itivity
Maska change at
Layer SM
Thaat acc
method
Pre-defined with frozen mask
N
89.72
36.60
Tayer 9-16
0.8
88.88
Without frozen mask
layer 1-16
layer 0.5
84.02
82.32
We train a separate model with the same architectare and target measure the final layer-wise sensitivity.
5

---

Published in Transactions on Machine Learning Research (09/2023)
Clest
SPOST mesh
INTAE dfferent sportty makk ovay tounds
to lowing sensioitt
Clerz2
Uypden vitted lnyer setuiiiity
Sendrenstviyy
weighe/
Globel spane upcete
corneteniation
Communication
Computation ovethedd
Perform once
Computation Laving
Repest times
Upcl-pane compune
Stage
spaise sompute
Stage
Figure 5: Summary of FLASH. Stage Ca clients perform sparse training locally find layer-wiste sensi- tivities. Stage 2: Clients collaboratively train the weights under the server's supervision to follow the layer sensitivities and reach consensus in yiekding sparse masks.
Based on this observation, we hypothesize that such a lack of consensus impacts the model's performance. Changing the mask every epoch means a large chunk of weights to be updated from 0, affecting the trained weights' maturity. We note that this an observation in a centralized setting. However, we believe it reflects generic phenomenon irrespective of the centralized or decentralized nature of training.
4 FLASH: Methodology
Based on obscrvations in section them find lottery licket individually, fails in finding global one. To help FL find the winning lottery; we identify two key characteristics of sparse learning, pruning sensitivity and mask convergence and present federated lottery aware sparsity hunting (FLASH) methodologies. To adhere to the identified feature, FLASH includes two stages: Stage that learns layer sensitivity to properly initialize the sparse mask; Stage that trains weights and masks. Below, we explain how each step works and highlight its importance. Algorithm details the training methods, and Fig. 5 summarizes FLASH and its components.
8 we hypothesize that deploying sparse learning in clients, though it helps
4.1 Stage 1: Mask Initialization
This stage aims to identify a good sparse mask the server can provide to the participating clients. However, since in FL, the server does not have access to the training data, the server requires the help of the clients to estimate layer importance. Server first randomly selects a small fraction of clients ([Cx) and asks them to perform sparse learning (locally) on their private data for a few warm-up epochs (Ex) (44 in Algorithm ! It then evaluates each of Ga clients' layer sensitivities via Eq. P For each layer I, the server estimates average density by averaging the sensitivity of that layer over ca clients, i.e. d' == Σe. di
di is the density at layer I in 7th client. As these averaged layer-wiss wise density values may not necessarily satisfy the target density (d) criteria, for model with K parameters, we present the following density re-calibration formulation
where
d. = d &. .y, where ry d x K Σ Et, &K
k is the dense model's parameter size for layer L. For each layer I in the model, the server creates a binary sparse mask tensor that randomly initialized with a fraction of ones x d= (79 in Algorithm 1
4.2 Stage 2: Seasitivity-Aware Training
Using Stage clients can start training with a sensitivity-driven initialized mask. However, this does not guarantee the mask convergence which we have shown (section 3. Obs. 2). Therefore, we propose the following two approaches that can significantly improve mask convergence.
Shhich the same as sensitivity for a layer.

---

Published in Transactions on Machine Learning Research (09/2023)
Algorithm 1: FLASH Training.
Data: Training rounds T, local epochs E, clients [Cv], clients per rounds Gr, target density d scnsitivity warm-up epochs Ed, density warm-up client ca. freez and Aggregation type Agr.
Oinit iniitRaddommsskkeeeiiittttt/
serverExecute:
# Calculate the laser-wise sensitivity in stage 1
Randomly sample clients [Ca] C [Cx]
for each client E [Ca] in parallel do
Oe + clientExecutee(@hnt Eg, 0) # freez =
S. f conputeSensitiiitt((..)
end
O" + i-nntsassssiiiiiiiiiiaaaaaaeeees
10 freez 1 if SPDST, 0 if JMWST
# Start Stage 2
for each round t 1 to do
13
Randomly sample clients [c,] C [CN] for each client C [c,] in parallel do
14
15
freez) 0 - <lieeeeEeeectte(ee---- end
16
17
O aggrParaatUpaattthsss((1111111 Agr)
18
O +- subsanpleServeeroodal (6)] freez)
19 end
20 return 0"
21 clientExecute(@pv E, freez)
22 for locol epoch i f-1 1 to E do
23
O₂e + dosprrreeeeeeeaaiggg.....,,.,., freez)
end
25 return O.
Approach 1: SPDST. In this approach, to achieve convergent mask and low values, the server pre-defines layer masks at initialization (set freez 1 at 410 in Algorithm D. This way, all the clients agree on the mask and only train the non-zero weights. This guarantees no mask divergence issue (sm( = 0 for all i, Moreover, as FLASH disentangles the sensitivity evaluation stage from the training, the pre-defined mask in this scenario benefits from the notion of layer scnsitivity. We thus aptly name this approach as derisen pre-defined sporse training (SPDST). Interestingly, earlier research (Bibikar let al. 2022) hinted at poor model performance with pre-defined masks, contrasting ours where we see significantly improved model performance, implying the importance of stage (as will be elaborated in section
Approach 2: JMWST. Here, after stage 1, the global mask is not frozen, and clients have the freedom to come up with their mask and change the global one (set freez = 0 at 410 in Algorithm D This way, the model masks and weights are jointly learned during clients' local learning, thus termed as joint mask weight sporse training (JMWST). However, as highlighted earlier, clients' naive sparse mask selection at the beginning of cach round costs a considcrable accuracy drop (scction 3 Obs. 1). To avoid this problem and increase mask consensus, JMWST allows the server to subsample sparse model with density d at round ++ 1 from the aggregated model at the end of round t.
As mentioned in section 3 with target density d, the aggregated model has density dserver > d. To enable efficient sampling of a sparse model while adhering to layer sensitivity, we leverage the density re-calibration strategy (Eq. 3) by taking the tth round's clients' average sensitivity into consideration (418 in Algorithm I The server performs magnitude pruning to retain the top-d fraction of paramcters for 7 layer and sends the pruned model to clients at round t ++1. 1. Intuitively, the server's sampling of non-zero weights reduces the chances of wasted updates and accelerates mask convergence due to alignment. with the layers' pruning
7

---

Published in Transactions on Machine Learning Research (09/2023)
sensitivity. Then, the next round's clients can perform sparse learning locally, yield another set of sparse models, send them to the server, and so on. By default, the server evaluates masks every round Tint = 1. However, it can increase the update interval, and clients update the mask after a specific Tint > round.
SPDST vs. JMWST. The update aggregation and communication are more straightforward in SPDST, as the position of the zero and non-zero parameters are fixed, and the global model's 's density remains at d both at the server and the clients. In terms of mask convergent, SM is always 0 for SPDST because the mask is pre-defined and does not change. In contrast, JMWST gives more freedom to the client, and they can participate in mask training. Additionally, we observed that JMWST has a lower SM than NST, and the mask it more convergent. For example, for the CIFAR-10 dataset after 300 rounds, the SM value for JMWST is ~85 lower than that of NST.
Extension to support heterogeneous density. The current framework assumes that all the clients train with the same target density d. However, users may have different resource limitations, and some may contribute more resources the training based on their budget. We now present helero-PLASH to support this eerourre-heterogenne
of users and adhere to their respective density budgets.
Let us assume there are N available support densities daet- = [d1,.., where d; < d:+1. Now, for hetero- SPDST and hetero-JMWST, we perform the same Stage 1 as explained before to create the masks for the clients with the highest density dy- For any other density d,, we sample a sparse mask from that with density d,+1. Note, while creating the mask from d++1 to di, we follow the layer-wise wise density re-calibration approach (Eq. 3). Similar to the original SPDST, after Stage 1, the server freezes all the N pre-defined mask for the heterogeneous case, and for the rest of the training, clients can use the mask associated with their budget. For hetero-JMWST, at the beginning of cach round, the server performs magnitude pruning to yield N sub-models meeting N different densily levels, contrasting to the creation of one model in JMWST. Participating clients of different densities use the corresponding sub-models to start sparse learning locally.
In hetero-FLASH, clients do not contribute to weights equally. Considering all the parameters from clients with sparse masks, specifically, the pruned parameters may hurt the updates that are not zero on that same position. Therefore, instead of FedAvg., the server performs aggregation we call Weighted Fed Averaging (WFA). In particular, with similar inspiration as Diao et al. (2020), instead of averaging over # of partic- ipating clients, we average each non-zero update by their total non-zero occurrences over the participating clients in round. We have provided the algorithm for hetero-FT.ASH in the Appendix.
5 Experiments
Datasets and models. We evaluated the performance of FLASH on MNIST (LeCun & Cortes 2010 on
MNISTNet [McMahan et al. 2017). FEMNIST on model described in (Caldas et al.) 2018 and CIFAR-10. CIFAR-100 Krizhevsky et al. 2009) and TinyImageNet PPuuransari & Ghili 2014) on ResNet18 (Further details the Appendix). For data partitioning of MNIST, CTFAR-TO, CIFAR- and TinyImageNet, we use Latent Dirichlet Allocation (.DA)(Redea et al. 2020) with three different a (a 1000, 1, 0.1). In this allocation, decreasing the value of the alpha increases the data heterogeneity among the clients. FEMNIST is built from handwriting 3400 different users (Han et al. 2020 making it inherently non-llD.
Training typerparameters. We use starting learning rate (Tinst) as 0.1 which exponentially decayed to 0.001 (Nend). Specifically, learning rate for participants at round I is т = Tima(exp( log( Rend ))) In all the sparse learning experiments, the pruning rate is 0 Other training hyperparameters can be found in Table D Furthermore, all the results are averaged over three different seeds.
Dinit
5.1 Experimental Results with FLASH
To understand the importance of stage in FLASH methodology, we identify bascline training with uniform layer sensitivity-driven pre-defined sparse training (PDST) in FT.. Table 3 details the performance of FLASH at different levels of d. In particular, as we can see in Table 3 columns 5 and 6, the performance of the model for both NST and PDST drops at ultra-low parameter density d = 0.05, 0.01. For example,
4Prune rate controls the fraction of non-zero weights participating in the prune-regrow during sparse learning.
8

---

Published in Transactions on Machine Learning Research (09/2023)
Table 3: Results for different datasets with FLASH (SPDST, and JMWST) and its comparison with NST and PDST for different densities (d E (5%, 10%) for all datascts and extreme sparse, d 1% for CIFAR-10, CIFAR-100 and TinyImageNer and data distributions (a {0.1, (,000),
Dataset
Data Distribution
Density (d)
Basellne
NST Acc %
PDST Acc %
SPDST Acc %
JMWST Ace %
JMWST(
Acc %
Ace %
98.79
mT (a 1000)
0.1
97.57 + 0.11
97.09 + 0.18
98.
0.06
97.95 + 0.16
98.00 + 0.16 97.37 0.23
0.05
95.19 1: 0.56
94.8 1: 1.04
97.46 0.14
97.24 1: 0.21
MNIST
non-TTD 1.0)
0.1
97.36 + 0.19
0.19
96.82 + 0.25
97.96
0.1.3
97. + 0.12
98. 11 0.12
0.05
95.73 0.31
95.34 1. 0.77
97.3 0.26
97.38 1. 0.11
97.59 0.07
0.17
non-HID 0.1)
0.1
96.19
0.22
94.41
1:23
97.22 0.43
96.53 1: 0.19
96.7 1: 0.14
0.05
91.66 :: 1.74
91.06 + 1.1
95.7 0.37
95.83 + 0.81
95. = 0.
1.0
88.56 0.06
ITD (a = 1000)
0.1
84.69
0:26
66.72:40.00
88 + 0.28
87.62 +: 0.35
87.96 0.13
0.05
0.54
86.87
0.01
52.70 1.17
70.17 0.70
82.35 + 0.11
83.59 =: 0.38
83.85 0 0.26
1.0
87.13 1 0.18
CIFAR-10
non-IID = 1.0)
0.1
83.16 0.19
85.07
86.42
0.19
86.15 = 0.31
86.36 0.13
0.05
75.1 0.76
83.33 0-0---0..0 0.14
85.64 0-0.8
85.34 +: 0.27
+ 0.24
0.01
50.71 1. 0.99
69.44 0.63
81.01 1. 0.50
82.38 1. 0.18
82.31 1. 0.12
1.0
77.61 0.49
0.1
71.18
1.23
74.82 0.72
76.74
1.46
74.74 + 1.07
75.47 1.18
0.05
61.29
2.76 42.66
72.32 1: 1.05
75.47 1. 2.31
73.9 1: 1.45
75.49 1: 0.9
0.01
0.50
59.30 0.36
70.61 + 1.82
68.80 + 0.62
71.21 1.98
1.0
65.38 + 0.27
IID 1080)
0.1
51.81
0.92
61.16 1 0.51
62.35 0.40
62.88 1. 0.26
62.89 0.24
0.05
56.67
0.25
0.01
22.64
0.75
1.16 38.99
49.67 + 0.44
51.53 + 0.76
51.81 0.13
1.0
65.17
0.27
CIFAR-100
non-TID 1.0)
0.51
60.87
0.40
62.13
0.26
61.50 +: 0.07
0.11
0.05
0.39 23.39
42.48
56.57
0.28
59.57
0.35
59.27 1: 0.62
59.85 0.35
0,01
0.37
38.99
003.99:1000 0.31
19.05 + 0.40
50.60 + 0.10
51.61 0
59.12 +
non-IID 0.1)
0.1
49.04
0.57
55.06
0.26
56.79
0.33
54.74 1. 0.68
55.54 0.71
0,05
37.33 2: 0.39
51.68 1: 0.32
54.344 :: 0.17
52.67 + 0.97
53.47 1: 0.
0.01
0.19 19:21
0.26 35.59
45.10
45.31 + 0.57
46.16 0.76
1.0
55.36 1 0.25
IID = 1000)
0.1
11.63
0.17
51.95 1 0.11
53.18
0.41
52.05 + 0.09
52.51 0.35
0.05
38.39
0.14
48.61
+ 0.25
51.31
0.41
50.37 + 0.48
50.53 0.54
0.01
20.50 0.43
37.85 1184511000 0.07
0.35 43.66
43.07 1. 0.80
44.41 0.17
1.0
54.76 + 0.35
Tiny Imtgeeeeeet non-IID
0.1
4-1.88
0.16
50.50
52.75
0.18
51.22 +: 0.30
51.81 0.08
0.05
38.03
0.27
47.52 1 0.29
$1.07
0.23
49.48
4: 0.23
49.76 à: 0.48
0.01
20.68 0.14
37.49 0: 0.52
42.91
+ 0.24
43.04 + 0.12
43.15 0.25
non-lID 0.1)
0.1
38.92 0.23
12.61 1:11::0000 0.
46.25
0.07
45.18 + 0.38
45.21 0.18
0.05
32.81 +: 0.06
0.43 40.69
0.443
44.56
0.10
43.31 + 0.19
44.22 0.16
0.01
0.13
33.79 0.15
37.80
0.06
37.32
38.32
1.0
84.68 0.20
FEMNIST
non-TIT)
76.92
0.42
76.01
1.26
1.000+0..00
83.02 + 0.21
83.4
0.05
37.33
0.39
51.68 118881100 0.32
54.34 0.17
52.67 1. 0.97
53.47 0.49
on CIFAR-10 (a= = 0.1), models from NST and PDST sacrifice accuracy of 16.35% and 5.32%, respectively, However, at comparatively higher density (d = 0.1), both can yield models with a lower accuracy difference from the baseline by around 6.46% and 2.82%. SPDST, on the other hand, can maintain close to the baseline accuracy at even ultra-low density for all data partitions and, interestingly, even outperform JMWST (rint 1) for the majority of the cases. These results highlight the efficacy of both sensitivity- driven sparse learning (as SPDST > PDST) and early mask convergence (as SPDST z JMWST) in FL sellings. Imporianily, for incrensed Tint in JMW, we observe a consislent improvement in accuracy. The inferior accuracy at Tint = 1 can be attributed to the mask divergence caused by frequent noisy gradient updates. We thus believe efficient hyperparameter search including Tint is essential for the sparse FL model's improved performance. Also note that JMWST requires additional communication of non-zero weight indices for rounds that masks updating, contrasting SPDST, with clients not. needing to send the mask at all,
9

---

Published in Transactions on Machine Learning Research (09/2023)
allowing to yield proportional communication saving the model density. Fig. 6 shows the accuracy vs. comm. round for the two proposed methods on different data distributions.
Extreme sparse learning. For CIFAR-10, CIFAR-100, and TinylmageNet, which are trained on a comparably more complex model (ResNet18), we also explore the extremely sparse training sce- nario where clients train only 1% of the weights (100x saving in communication). This experiment shows the power and importance of both stages in FLASH in achieving good performance. In contrast, the performance gap between the dense model and NST PDST remarkably increases.
SP35110-0.1
sresta=1)
5P051ta=1000)
Comparison with Ze Despite leveraging a form of sparse learning [Raihan & Aamodi 2020), ZeroFL re- quires a higher up-link/down-lingg budget than the tar- get density d achieve better performance. However, FLASH reaches considerably better accuracy while being substantially more efficient in communication compared to ZeroFL. Especially for SPDST with a frozen mask, the server and clients only need to communicate updates with density and enjoy bidirectional savings. Note that every sparse matrix representation requires the value and loca- tion of non-zero parameters, but SPDST can eliminate sending the latter as it does not change. Finally, we evaluate the communication saving as the ratio of the dense model size and corresponding sparse model size represented in compressed sparse row (CSR) format (Tinney & Walker 1967). As depicted in Table [ FLASH can yield an accuracy improvement of up to 10.1% at a reduced commmication cost of up to 10.26x (at up-link when both send sparse models).
100071e-1000)
d-0,00 00.5
d-0.1
Test Accusacy PN4
Accureccy 1960
60-
20-
10
Figure 6: Test accuracy vs. round for CIFAR-10
with (a) d 0.05 (b) d= 0.1.
Table 4:
Comparison with ZeroFL on various performance metrics. (ZeroFL values are the results with the higher accuracy and taken from the original manuscript.)
Dataset
Data
Method
Density
Acc%
Down-link Savings
Up-link Savings
Distribution
ZeroFL. QQu
2021
82.71 0.37
1.6x
SPDST (ouzs
0.28
9.8x
9.8x
IID
ZeroFL
Quu
2021
0.05
78
0.55
1.9x
DSP (ours
0.05
86.99
0.14
19.5x
19.5x
CIFAR 10
ZerolFT. QQiu al.
2021
0.1
81.04
+ 0.28
1x
1.6x
non-IID (a 1.0)
SPDST [ours]
0.1
86.42
0.49
9.8x
9.8x
ZeroFL [Quu et al.
2021
0.05
75
+ 1.15
1x
1.9x
SPDST (ours)
0.05
85.64
+ 0.58
19.5x
19.5x
ZeroFL. [Qiu et al.
2021
0.05
77.16 + 2.07
1x
17.7x
FEMNIST
IID
SPDST (ours)
0.05
81.18 0.36
14.6x
14.6x
5.2 Experimental Results with Hetero-FLASH
Table 5 shows the performance of hetero-FLASH for the scenario where the clients can have three possible density budgets defined by the daet with maximum clients' density dmaz = 0.2. Also, we assume 40%, 30%, and 30% of total clients can train models with a density equal to 0.2, 0.15, and 0.1, respectively. The server samples 10% from each set for every round with the corresponding target density. Similar to the trend FLASH, hetero-SPDST outperforms the hetero-JMWST (Tint = 1), and increasing mask update interval (rane) helps improve its performance (roughly > 3%).
5.3 Quantitative Analysis on FLASH's Design Parameters
Impact of initial sensitivity warm-up of participating clients. For realistic scenario, Stage needs to be efficient and practically feasible. To be more precise, we cannot expect participating clients to train
We understand for FEMNIST, ZeroFL reported significantly higher up-link saving; however, best of our understand- ing, it should be similar their report on other datasets, i.e. 11.x.
10

---

Published in Transactions on Machine Learning Research (09/2023)
Table 5: Performance of hetero-FLASH on various datasets where each client can have a density from the set dee E [0.1, 0.15, 0.2] based their budget, Note that the density of the final model depends on dmaz-
Dalasel
Distribution
Max daet
Heeero-SPDSTT
Helero-JMWST (ritt Ace
Holsro-JMWST
Acc: %
(Tiu 5) Acc %
IID (a 1000)
98.29 + 0.05
97.41 = 0.23
97.83 0.10
MNIST
non-Iil) [a 1.0)
0.2
98.29
0.09
97.47 +
97.80 0.23
non-IID (a 0.1)
97.63
0.22
96.11 0.75
96,25 0.86
IID (a 1000)
87.19 + 0.26
86.37 0.2
87.39 + 0.15
CIFAR-10
non-III (a 1.0)
0.2
86.16 + 0.04
84.67 = 0.06
86.19 0.24
non.IIll (a 0.1)
75.23 1.26
71.3 2.75
74.34
IID (a = 1000)
63.4 4
60.5 + 0.8
62.91 # 0.04
CIFAR-100
non- mon-1II IID [a 1.0)
0.2
62.1 1: 0.2
59.6 2: 0.2
62.68 1: 0.28
non-lIl [a 0.1)
56.4 0.4
51.5 0.9
55.98 + 0
II (a 1000)
51.28 1: 0.11
49.73 1: 0.20
52.07 1 0.22
Tiny Net
non III) (a 1)
0.2
50.94 4: 0.17
49.32 1: 0.16
$1.61 4: 0.17
non-III) [a 0.1)
4 0.40
42.72 +: 0.61
+ 0.45
FEMNIST
non-lll)
0.2
82.58 + 0.24
82.2++
82.5 + 0.55
their models for long time in order to get an accurate estimation of layer sensitivity. Also, it is unlikely to access many clients in a single time slot, especially in cross-device federated learning. Therefore, we designed six different scenarios to understand the impact of these parameters on the final model's performance. In particular, we used two different values of participating clients ([10, 20)) and three local epoch choices ([10, 20, 40)). As shown in Fig. [a), the yielded pruning sensitivity follows a similar trend. Moreover, SPDST with a mask chosen from any of these sensitivity lists finally yields FL models with similar performances (Fig. (b)), clearly demonstrating the robustness of our warm-up based sensitivity evaluation stage 1.
Eg- 4p=10
1,- 20-20
20, cy-10
Eg-20, 2y-20
Ey-40, 6y=10
E,-40, 2
Resher188 OIAR-10 d=0.00)
ResNit18
d=0.05
Restie18
d=0.05)
Srusing Secshaity
Test Accurccyyyyy
80
Tost Acourscy
86
60
80
64-
20-
SPOST
JMWST
200 100
300
400
100
200
300
400
Conv
Index
Comm. Round (c)
Commm Round (1)
Figure 7: (a) Layer sensitivity evaluated at the end of sensitivity warm-up stage (Stage 1) for different client participation (ca) and their local epochs (Ea); (b) Comparison of global model performance with the initialized sparse mask based on different sensitivity evaluated from (a).
Overheads of stage Stage uses one round with Eg local epochs (here, Es = 10) per client. A normal FL stage in our settings trains the clients for T rounds, 1 epoch per client /round. Hence, this stage increases the time by a factor of ( E + 1). Usually, F.a << T, making the pre-training overhead negligible.
The communication overhead of Stage 1 is also negligible compared to that in each round for the Stage 2 FT. training. Fach participant only needs to send L values for an L-layer model. So, ca clients will have a lotal communication overhead of (L Ca x 32) bits, assuming b2-bit number represen- tation.
Reclette on CITAR-10 0.05 am
Restlett on CFAR 00-000000000 20, 0.
200
FOST
feer SPOST
Teckig
Communication
Communccation Cost (C2B)
Convergence versus communication costs. Fig. S shows the performance FL models when Figure 8: Performance vs. up-link bandwidth for (a) = 1.0 and (b) a 0.1.
the clients have a limited communication budget. In particular, PDST and SPDST can significantly outperform other approaches at low communication budgets (even FedAvg). This be attributed to
11

---

Published in Transactions on Machine Learning Research (09/2023)
their substantially smaller model sizes, helping them to communicate more rounds than others limited bandwidth scenario.
Importance weighted aggregation in hetero-FLASH. Earlier literature (Diao et al. 2020 suggested weighted averaging in the aggregation models with different sizes, which we also investigate here.
particular, we performed experiments on CTFAR-10 (a 1.0), both with and without WFA. First, we observe that WFA degrades model performance in JMWST compared to FedAvg (Fig. g (a)). On the contrary, the of WFA improves accuracy for Netero-FLASH (Fig, g (b)). The inferior performance of WFA in FLASH may hint at the fact that if parameter is non-zero only for fewer clients, as compared to other non-zero weights, giving it equal weight as the others in the aggregation mullifies its lower importance, that may be necessary to preserve for mask convergence. On the other hand, having WFA in hetero FLASH is necessary, as the less frequent non non-zero occurrence of a parameter can be a result the presence fewer high-parameter density clients a round.
100
Bes'eet28 CIPAn-3 11
Tesheet5 on CITAB-10 (haters SPOST)
:0-
Reshetts on CHRaR 10 05)
Resheetl on CIFAR-ID (d=0.1)
$7.19
AVN
recurecy
13-
mic
Meig Packeg
Algxille
Data Pattitun le)
Data Partiton [o]
Data Partiion (e)
Figure 9: Performance comparison between FedAvg and weighted FodAvg for different (a) algorithms (b) data distributions (a)).
Figure 10: Performance models trained with dif- ferent mask initialization in stage 1 for target pa- rameter density for (a) d 0.05 (b) d= 0.1
Comparison with ERK+ initialization. We now compare our SPDST mask initialization with that of parameter density distribution evaluated via ERK+ (Huang et al. 2022 Evei et al.) 2020). In contrast with uniform density, the ERK+ scheme keeps more weights for the layers with fewer parameters. To this aim, we use Stage SPDST, ERK+, or uniform (PDST) as the initial mask for stage and keep the mask frozen for the rest of the training. As shown in Fig. 10 the mask initialization using stage for SPDST consistently provides superior results over the other two. We hypothesize this is rooted in the data-driven layer sensitivity evaluation scheme of SPDST, particularly at the earlier layers, allowing it to retain more information at these layers.
Ablation on Stage 1. Our proposed method consists of two stages: sensitivity evaluation (Stage 1) and training in federated settings (Stage 2). In Table G we present ablation with and without Stage 1 for SPDST and JMWST. It is notable that SPDST without Stage PDST.
Table 6: Impact of Stage 1 final performance CIFAR-10 dataset with target d = 0.05
Data Distribution
Method
without Stage
with Stage
IID 1000)
SPDST
84.38 0.12
86.99 0.14
non-IID(a 0.1)
SPDST
72.32 1.05
75.47 :: 2.31
IID 1000)
JMWST
0.1
87.18 0.09
non-IlD(a 0.1)
JMWST
74.7 1.7
75.49 : 0.9
6 Conclusion
This paper presented methodologies to yield sparse server models with insignificant accuracy drops pared to the unpruned counterparts. In particular, we demonstrated two efficient sparse lcarning solutions specifically failored for FL, enabling better computation and communication benefits over existing sparse learning alternatives, Additionally, we presented the effectiveness of the proposed algorithms for clients with different parameter budgets, allowing deployment for resource-liitted edge devices having heterogencous resource support. The future research direction of this work includes theoretical understanding of our observations and further empirical demonstrations of the newer class of foundation models.
12

---

Published in Transactions on Machine Learning Research (09/2023)
7 Acknowledgement
This material is based upon work supported by ONR grant N00014-23-1-2191, ARO grant W911NF- 22-1-0165, Defense Advanced Research Projects Agency (DARPA) under Contract No. FASTNICS HR0011200000 and HR00112000160) and gifts from Intel and Qualcomm. The views, opinions, and/or /or findings expressed are those of the author(s) and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.
References
Sameer Bibikar, Haris Vikalo, Zhangyang Wang, and Xiaohan Chen. Federated dynamic sparse training: Computing less, communicating less, yet learning better. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 6088, 2022.
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Koneōný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.00 01007, 2018.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing perfor- mance. XrXiv preprint arXi::1907.0484/ 2019.
Jacob Devlin, Ming-Wee Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprint arXiv:1810.04800 2018.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010. 012. 2020.
Xiaohan Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, Ji Liu, al. Global sparse momentum sgd for pruning very deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Ctku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943 2952. PMLR, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. Xiviv preprint vrXic:1803.03633 2018.
Eduard Gorbunov, Konstantin P Burlachenko, Zhize Li, and Peter Richtárik. Marina: Faster non-convex distributed learning with compression. In International Conference on Machine Learning, pp. 3788-378 3798. PMLR, 2021.
Pengchao Han, Shiqiang Wang, and Kin K Leung. Adaptive gradient sparsification for efficient federated learning: An online learning approach. In 2020 IEEE 40th International Conference on Distributed Com- puting Systems (ICDCS), pp. 300-310. IEFE, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Tlias Leontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 34, 2021.
Tiansheng Huang, Shiwei Liu, Li Shen, Fengxiang He, Weiwei Lin, and Dacheng Tao. Achieving personalized federated learning with sparse local models. ariie preprint orXiv:2201. 11380, 2022.
Berivan Isik, Francesco Pasc, Deniz Gunduz, Tsachy Weissman, and Michele Zorzi. Sparse random networks for conmmiicttion-efffieen federated learning. ICLR, 2023.
Yuang Jiang, Shiqiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K Leung, and Leandros Tassiulas. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems, 2022.
13

---

Published in Transactions on Machine Learning Research (09/2023)
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Hu- bert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang Hc, Lie Hc, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konecný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Pratock Mit- tal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Hang Qi, Daniel Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Fe- lix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. Foundations and Trendoo in Machine Learning, 14(1 2):1 210, 2021. ISSN 1935-8237. doi: 10.1561/22000 /20002200000083. URL http: //dx://dx.wo doi. org/10..111... 100.0.5511222200 1561/2200000883
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Souvik Kundu and Sairam Sundaresan. Attentionlite: Towards efficient self-altention models for vision. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 2225-2229) IEEE, 2021.
Souvik Kundu, Saurav Prakash, Haleh Akrami, Peter A Beerel, and Keith M Chugg. psconv: A pre- defined sparse kernel based convolution for deep cnns. In 2019 57th Annual Allerton Conference Communicatiinn, Comtrol, and Computing (Allerlon), pp. 100 107. IEEE, 2019.
Souvik Kundu, Mahdi Nazemi, Massoud Podram, Keith M Chugg, and Poter A Becrel. Pre-defined sparsity for low-complexity convolutional neural networks. JEEE Transactions on Compulers, 69(7):1044 1058, 2020.
Souvik Kundu, Mahdi Nazemi, Peter Beerel, and Massoud Pedram. Dur: A tunable robust. pruning framework through dynamic network rewiring of dnns. In Proceedings of the 26th Asia and South Pacific Design Automation Conference, pp. 344-350, 2021.
Souvik Kundu, Yao Fu, Bill Ye, Peter A Beerel, and Massoud Pedram. Toward adversary-awar non-iterative model pruning through d ynamic n etwork r ewiring of dnns. ACM Transactions on Embedded Computing Systems, 21 (1) 24, 2022.
Souvik Kundu, Shunlin Lu, Yuke Zhang, Jacqueline Liu, and Peter Beerel. Learning to linearize deep neural networks for secure and efficient private inference. International Conference on Learning Representation, 2023a.
Souvik Kundu, Sairam Sundaresan, Massoud Pedram, and Peter A Beerel. Float: fast learnable once- for-all adversarial training for tunable trade-off between accuracy and robustness. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2349 22589,238.--- 2023b.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http::/yann. lecun.
com/exde ht///exdb/mmis///
Ang Li, Jingwei Sun, Binghui Wang, Lin Duan, Sicheng Li, Yiran Chen, and Hai Li. Lotteryfl: Personalized federated learning with lottery ticket hypothesis on non-iid datasets. arXiv
and communication-effieeeee preprint ar Xrx:22008. 03371, 2020a.
Ang Li, Jingwei Sun, Xiao Zeng, Mi Zhang, Hai Li, and Yiran Chen. Fedmask: Joint computation and commuication-efficiantt personalized federated learning via heterogencous masking. In Proceedings of the 19th ACM Conference on Embedded Nebworked Sensor Systems, pp. 42 55, 2021.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 14129-450, 2020b.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arxiv preprint ar 109.00190, 2019.
14

---

Published in Transactions on Machine Learning Research (09/2023)
Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, and Mykola Pechenizkiy. Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware. Neural Computing and Applications, 33(7):2589 2604, 2021.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning- arviv preprinl or Xrv: 1ate:111000000 510.05270, 2018.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Commmicationn-fficint learning of deep networks from decentralized data. In Artificial intelligence and slatistics, pp. 1273 1282. PMLR, 2017.
Yiqun Mei, Pengfei Guo, Mo Zhou, and Vishal Patel. Resource-adaplivve lederated learning with all-in-one neural composition. Advances in Neural Information Processing Systems, 35:4270-422 42000, 2022.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madcleine Gibeseu, and An- tonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature commanscalions, 9(1):1-12, 2018.
Yue Niu, Rajgopal Kannan, Ajitesh Srivastava, and Viktor Prasanna. Reuse kernels activations? flexible dataflow for low-latency spectral cnn acceleration. In Proceedings of the 2020 ACM/SIGDA International Symposium on Field- Programbogee Gale Arrays, pp. 266 276, 2020.
Yue Niu, Saurav Prakash, Souvik Kundu, Sunwoo Lee, and Salman Avestimehr. Federated learning of large models at the edge via principal sub-model training. arXiv preprint arXim:2208.131 13141, 2022.
Hadi Pouransari and Saman Ghili. Tiny imagenet visual recognition challenge. CS231N course, Stanford Univ., Slanford, CA, USA, 5, 2014.
Eric Qin, Ananda Samajdar, Hyoukjun Kwon, Vineet Nadella, Sudarshan Srinivasan, Dipankar Das, Bharat Kaul, and Tushar Krishna. Sigma: A sparse and irregular gemm accelerator with flexible interconnects for dnn training. In 2020 IEEE International Symposium on High Performance Compuler Archilecture (HPCA), pp. 58 70. IEEE, 2020.
Xinchi Qiu, Javier FernandezMarqqes, Pedro PB Gusmao, Yan Gao, Titouan Parcollet, and Nicholas Donald Lane. Zerofl: Efficient on-device training for federated learning with local sparsity. In International Conference Learning Representations, 2021.
Md Aamir Raihan and Tor Aamodt. Sparse weight activation training. Advances in Newral Information Processing Systems, 33:15625 15638, 2020.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's hidden in randomly weighted neural network? Tn Proceedings of the IEEE/ /CVF Conference Computer Vision and Pattern Recognition, pp. 11893 11902, 2020.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koncčný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprinl 20Xi0:2003.00295, 2020.
William F Tinney and John W Walker. Direct solutions of sparse network equations by optimally ordered triangular factorization. Proceedings of the IEEE, 55(11):1801 1809, 1967.
Hang Xu, Chen- Yu Ho, Ahmed M Abdelmoniem, Aritra Dutta, El Houcine Bergou, Konstantinos Karatseni- dis, Marco Canini, and Panos Kalnis. Compressed communication for distributed deep learning: Survey and quantitative evaluation. Technical report, 2020.
Giyoung Yang and Taewhan Kim. Design and algorithm for clock gating and flip-flop co-optimization. In 2018 TEEE/ACM International Conference on Compuler-Aides Aided Design (ICCAD), pp. 1 1-6. IEFE, 2018.
15

---

Published in Transactions on Machine Learning Research (09/2023)
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint Xri:: 19957, 2019.
Yuchen Zhang, John Duchi, Michael T Jordan, and Martin J Wainwright. Informatiinnthheoreicc lower bounds for distributed statistical estimation with communication constraints. In C.J. Burges, L. Bot- tou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Aduances in Newral Informalion Processing Systems, volume 26. Curran Associates, Inc., 2013. URL httpss:/ppoceeeding neurips. cctppipppccppppe///////pp pdf
Aojun Zhou, Yukun Ma, Jumnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hong- sheng Li. Learning fine-grained structured sparse neural networks from scratch. orXiv preprint arXiv:2102.04011 04010, 2021.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. In International Conference on Machine Learning, pp. 12878 12889. PMLR, 2021.
16

---

Published in Transactions on Machine Learning Research (09/2023)
A Appendix
A.1 Hetero-FLASH Algorithm
Algorithm 2 details the training algorithm in hetero-FLASH. Note that and aggrParanUpdatteeskk and
subSamppleServerModel the two functions that play key role in supporting heterogencity in sparsity ratios for different clients. The details of these two functions are elaborated in Algorithm and Algorithm respectively. We plan open-source our code upon acceptance of the paper.
Algorithm 2: Hetero-FLASh Training.
Data: Training rounds T, local epochs E, client set [[Cx,], ..., [CNaell clients per rounds cry target density set dget .d....., darl, sensitivity warm-up epochs Es, density warm-up client count cá, initial value of freeze masks freez = O, training algorithm A and aggregation type Agr.
fmst smiitBaandommaskkeeeeeeetdddddd
serverExecute:
Randomly sample ca clients [Ca] [CN for each client E [Ca] in paraltel do clientExecute((mm) Ea,
S. + compute@ensitiiitt(((...
end @ -it.tteuuutttttttraaaaaaaaaaaaae
freez freezeMask(A)
10 for each round t to T'
Randomly sample clients [Cr] C [Cw]
12
for each cliend E [C.] in parallel do
13
O + ===ccleetxeecuee(""" end
freez)
14
15
O/ +
aggrParanUjpdatteaas (105) Agr)
16
+
subSumpleServerMooeel (0's, (0%], duet freez)
end
18 clientExecute(Op.. freez):
for local epoch i E do
20
O, + dosparseleeariing@9 1, freez)
21 end
return O_F
Algorithm 3: aggrParranUpdateeMaak
Data: Round i, aggregation type Agr [fedAvg, weightedFedAvg) clients updates [0'] = [0,, .... Oc.], client
data size [dsc]....,dde ds.,]
if Agr is fedAug then
O So does D dsc, O
else
//For FLASH
W' +- -initieighFFactor())
for each update O., [ do
W + dse, rotrieveMasx(O.)) W W" Wt
end
10
//safeDivide(a.b) gives zero anywhere the b is equal to zero 0 - - | (sfeDDvvdddWWW W*) 0%,]
end
17

---

Published in Transactions on Machine Learning Research (09/2023)
Algorithm 4: subsamppeeServer/oooell
Data: Current round id f, client set [C-], aggregated Weight 0' of model with layers, support density set daot [[d,..., dar] where d, d++1; model layer-wise parameter count [k. ,.... k").
if (ize( is then
//JMWST subsampling in FLASH
M - initMaskMithZeerosl/
[d' d=j + avglayerviisaaeaait((((....
Ts Σ &
for layer 1 + to L do
7
idx + getSbrreeWeeigghttnnniesee...
n. - ini((r x d' x k)) // //mumber of non-zeros
8
M'(iixl|: nz]] 1
9
10
end
else
12
//For Petero-FLASH
13
for d: E deat do
14
M; t- initWaskWittherrss)
15
end
16
D - (ettaurrreenc)))sstt(((()))
[².....d]] +- getlayerriisaDemstty(88. for layer < 1 to L do
18
idx getBrteelWeegghnnddceeeee.
19
20
for d, E dae: do - de rr.
21
Dy
n, int(r;; x d' x k')
M, [idx[ nz]]
24
end
25
end
28
end
A.2 Model Architectures
Table 7 shows the model architectures used for MNIST and FEMNIST datasets. For CIFAR-10, CIFAR-100
and TinyImageNet we used RosNet18 (He et al. 2016 with the first CONV layer kernel size as x 3 instead of original 7 x 7.
Table 7: Architecture used for MNIST and FEMNIST datasets
MNIST
FEMNIST
CONV5 x 5(C. 32) CONV5 x 5(C. = 10)
max pool
max pool
CONV5 x 5(C. 20)
CONV5 x 5(C. 64)
max pool
max pool
FC(5120, 50)
FC(3136, 2048)
FC(50, 10)
FC(2028, 62)
A.3 Additional Comparisons
We now compare the performance of FLASH with that yielded via FedSpa (Huang et al.) 2022) and FedDST (Bibikar et al., 2022). For FodSpa, we implemented their proposed algorithm our settings and kept all the hyperparameters the same for an apple-to-apple comparison. We report the best accuracy yielded for FLASH via models trained using SPDST and JMWST. As shown in Table FLASS outperforms FedSpa
18

---

Published in Transactions on Machine Learning Research (09/2023)
up to 2.41%. A similar trend is observed when compare with FedDST, and as Table 9 dataset, FLASH can have an accuracy improvement of up to 1.41%.
the MNIST
Table 8: Comparison of FLASH with FedSpa
(Huang et al.
2022
CIFAR-10 with ResNet18.
Data distribution
Method
Density (d)
Best Acc. (%)
A
a 1000
FedSpa
0.05
85.63
FLASH
0.05
87.18
+1.55
a = 0.1
FedSpa
0.05
73.08
FLASH
0.05
75.49
+-2.1
Table 9: Comparison of FLASH with FedDST Bibikar et al. 2022 on pathologically non-IID MNIST. We used the same hyperparameter settings and models as (Bibikar et al.
2022 for this comparison.
Method
Density (d)
Communication Best Acc. (%) Cost (GiB)
ô A
FedDST
0.2
96.10
FLASH
97.51
+1.41
F
0.2
2.0
97.35
FLASH
97.69
+0.34
B FLASH for NLP Tasks
We also employed FLASH
fino-tuning the BERT-basc (Devlin al. 2018), a popular large language model that can be trained comprehensively with academic resources. In this experiment, the embedding layers are frozen, and since the goal is to fine-tune the model, the total number of federated rounds is 50. As shown in Table 10 the improvement in SPDST/JMWST over the baseline alternatives can also be observed on NLP tasks.
Table 10: Results for fine-tuning SST dataset on BERT-base model.
Dataset
Distribution
Density (d)
Dense
NST Ace %
PDST
SPDST
JMWST(rint 1)
JMWST 5)
Ace %
Ace
Ace
Acc %
%
SST2
non-IID (a
0.2
92.25
77.59
78.96
83.07
80.5
81.78
C More Quantitative Analysis
Below, we provide more analysis and ablation to show the effectiveness of FLASH in different scenarios.
C.1 Impact of Number of Participating Clients per Round
Fig. (a) shows that JMWST and SPDST follow the same pattern at the baseline model (d = 1.0) with FedAvg. In other words, similar to FedAvg, as the o increases, the performance enhanced. Also, for a specific 6, JMWST and SPDST perform better than PDST and NST.
C.2 Impact of Batch-Normalization Layer Statistics
Fig. (b) shows the performance comparison between batch normalization (BN) and static batch normal- ization (static BN, as suggested in (Diao al. 2020)). particular, in our setting, using BN layer statistics consistently outperforms the static BN.
19

---

Published in Transactions on Machine Learning Research (09/2023)
100
ResNet18 on CIFAR-10 (a= 1)
ResNet18 on CIFAR-10 (a 1)
100
Test Accuracy (%)
90
Test Accuracy (%)
90
87.13
85:15
85:34
82.48
85.64
70-
70
NSTId-0:051
50
POSTTd=0.05)
50
SPDSTid=0.00 05)
[MW5T]6=D.05)
40
30
FodAug (d-1)
static
5
10
20
20
FedAvg
JMWST
SPDST
#Clients per Round (C,)
(d=1)
(d=0.05) Method
(dm0.05)
(a)
(b)
Figure 11: (a) Performance of the final trained model for different participating clients per round, (b)
Significance of BN and Static BN in final model performance.
C.3 Effect of the Mask Update Interval Rounds (Fine) in JMWST
As mentioned in the original manuscript, for JMWST, the server can increase the mask update interval (Féne) to save communication energy. We thus performed ablation for this variable from the default value of (similar to (Qiu al. 202Th) to its impact on the final accuracy, and Table 11 and Fig. 12 show the results. In particular, as can in the table, less frequent update intervals can lead to better performance than the original JMWST and provide additional bidirectional saving in communication as the masks do not change every round. Fig. 12 also indicates that the improvement tends to saturate after specific Tint, which hints at the importance of this paramcter. This pattern in the performance means that Tint may potentially create trade-off between the learnability of the masks and weights, and we believe understanding this complex trade-off an interesting future research.
Table 11: Impact of different mask update intervals in JMWST for a target density d = 0.1 on CIFAR-10.
Model
Data distribution
Mask update interval rounds (rea)
Fint
Tint = 2
Fedt
Fiut 10
IID (a 1000)
87.62 0.35
87.76 = 0.07
87.86 0.13
87.67 0.09
ResNet18
non-lID (a
86.45 0.31
86.26 1 0.07
86.36 1 0.13
86.68 +: 0.25
nou-IID (ax = 0.1)
74.74 1.07
73.73 1 1.18
75.47 1.08
77.14 1 0.22
ResNet18 CIFAR-10 (d=0.05)
Test Accuracy (%)
85
80
75
70
- a 0.1
-O-- a=1
65
12
5
10
20
Iint
Figure 12: Test accuracy vs. mask update interval round.
C.4 Convergence Trend Proposed Algorithms
Fig. 13 shows the test accuracy vs. FL rounds for NST, PDST, SPDST, and JMWST algorithms on the
CIFAR-10 datasct with non-IID data distribution (a = 1). As shown the plots, for d = 0.05 and d 0.1, NST has slower convergence with lower final accuracy. Introducing consensus among the clients for the sparse mask accelerates the convergence and enhances the final performance.
20

---

Published in Transactions on Machine Learning Research (09/2023)
100
100
80
80
Mal Accunac (h1)
Accuncc
NST
NST
PDST
PDST
20
20
JMWST SPDST
SPOST
100
200
300
400
500
600
100
200
300
400
500
600
(a]
Comm. Round (O
1b)
Cornn Round
Figure 13: Performance of proposed algorithms vs. comm. rounds on CIFAR-10 dataset for (a) d 0.05 (b) d 0.1.
C.5 Revisiting Sparse Mask Mismatch for NST with VGG16
Fig. shows the comparison of SM between centralized and FL settings with NST on VGG16, another popular model variant. Similar to our observed trend with ResNet18, we see significantly high SM for FL scttings with target of d = 0.05. This strengthens the generality of our observed limitations across different classes of DNN models.
VOGIe DINN 10 10HO Controlloos
V0036 CPPAR-20 10-20000 20 Naderatal)
119
(9)
Epech
[c)
Com Rourd
Figure 14: (a)-(b) Sparse mask mismatch (SM) for VGG16 in (a) centralized and (b) FL scttings with NST. (c)-(d) Layer-wise SM vs. training epochs (rounds) for VGG16 in (c) centralized and (d) FL settings, respectively, with NST.
C.6 Revisiting Sparse Mask Mismatch for FLASH
As demonstrated in Fig. 15 the sparse mask mismatch in the case of JMWST significantly reduces, helping the mask train in convergent way, significantly faster than that in NST.
Fig. shows the layer-wise SM for the centralized trained model (Fig. 16a) and FL trained model with sparsity (Fig. 16 c-). In particular, the SM at the later layer can significantly reduce in the case of JMWST compared to NST, further demonstrating the convergence ability even at the later layers.
ResNet18 on CIFAR-10 (d-0.05, Centralized),
ResNet16 on CIFAR-10 dw0.05, Federated!
ResNec18 on CIFAR-10 la d=0.05, Federateg)
Camin Bound
75-
75-
Epoch
125
125-
175
175
225
175
275
8
275
(a) Epoch
(b) Comm, Round
[c)
Comm.
Round
Figure 15: Sparse mask mismatch (SM) for (a) contralized sparse learning, (b) NST, and (c) JMWST in lederated settings.
21

---

the Published in Transactions on Machine Learning Research (09/2023)
Rechiet18 OFAR (5-0 Contralaed
RosHeti8 CIARR11 to 8-0.5 Federited)
ReshettS CPAR le d.0.05 Foserated:
Towards mask agreement layers
lapers
Significanlly
0.
jecerd Disaance
0.4
0.4
1.4
000
25
50
[3]
103
125
150
175
200
0.0
90
(b)
750
2013
250
300
0.0
(c)
Com 1o Round
103
200
Figure 16: Layer-wise sparse mask mismatch (SM) vs. training epochs (rounds) plot for (a) centralized and (b) FL with NST, and (c) FL with JMWST.
C.7 Sparse Mask Mismatch Function of d
To understand the relation of SM with d, we performed the baseline sparse training (NST) with ResNet18 CIFAR-10 for three different target densities, 0.05, 0.25, 0.5. As shown in Fig. 17 the SM tends to reduce for higher density. In particular, Fig. d7(d) shows the SM for CONV layer 16 (a later layer) after round 200. The SM reduces by 1.53x for d = 0.5 than that with 0.05, strongthening our general observation that SM becomes prominent as the density decreases.
Restltt8 OFAR Ip d=0.05 Reslltt18 CIFAR330 (p Federsted)
ResNiet0 CIRAR dio.5. Federated)
Bestlttt CIAA Ityee d=16, Round-2000
Corm Bound
125-
44
325-
2.52
175-
Corm
115-
225
225
4
1
Comm.
Round (4)
Comm. Round
Comm Round (4)
Dersity
(d)
Figure 17: (a-c) SM for FL settings for three different of 0.05, 0.25, and 0.5, respectively. (d) Comparison of Jaccard distance values for the 16th CONV layer of ResNet 18 after round 200 for different ds.
C.8 Sparse Mask Mismatch as a Function of Total Number of Clients
To understand the relation SM with the number of total clients, we performed the baseline sparse training (NST) with 18 on CIFAR-10 for 50 and 200 clients, respectively. As shown in Fig. 18| the SM concern persists, irrespective of the number of clients. This strengthens the generality of our observations total number of clients.
Pro/Au320 CIRR F-1--2 fprm 05. tedorate!
Mushvet2 CIAI g. haderaiaf!
1.0
PaoAAeE3 CP C2000000
redurated!
Coven Sleedt
22
122
Cover. found
Conre:
faund
Commi found (6)
Domer found
(6)
Figure 18: (a-b) SM for FL settings for (a) 50 and (b) 200 clients. (c-d) Layer-wise SM vs. training rounds for (c) 50 and (d) 200 clients.
D Model Personalization vs. Mask Consensus
Compared to personalized FL, FLASH comes with a different objective. Our goal is train single global model that performs well all clients' datasets. Mask convergence plays key role in maintaining stable
22

---

Published in Transactions on Machine Learning Research (09/2023)
learning progress and high model accuracy, especially for heterogeneous data and high compression ratios. However, model personalization aims to find individual models that adapt to each client.
E Discussion on Computing Benefits of Sparse Learning at the Edge
To extract FLOPs benefits for irregular pruning in FLASH, we assume that the compute energy for the sparse network can be avoided via the means of clock (Yang & Kim 2018) of the zero-valued weights. Moreover, there has been recent development of sparsity- friendly DNN accelerators (Qin et al. 2020) that can efficiently reduce the compute cost by significant margin. Such accelerators can leverage the yielded sparse FL models to deploy at compute-consstaiined edges.
E.1 FLOPs vs. Communication Cost for Different Density Budgets
To reach target accuracy value, we plot the FLOPs to uplink communication cost for different density budgets in Fig. 19
ResNet18 CHAR-11
ResNet18 CIFAR-10
ResNet18 CIFAR-10
0.5
40
FLOPs (GMAC
FLOPS (GMAN
cost (MB)
30
20-
0.2
0.1
0.1
0.0
0.0
0.0
0.8
(a) Comm cost (MB)
(b) density (d)
(c) density (d)
Figure 19: Computation and Communication relation with (a) each other (b, c) with different density levels
for SPDST algorithm.
E.2 Clarification on Sparse Training vs. Sparse Updates
Here, we want emphasize that our framework differs from the method proposed for sparsifying updates such top-K. In particular, in algorithms that try to sparsify the updates, the clients must do dense gradient update, costing higher computation and potential communication overhead. Our framework, specifically SPDST, on the contrary, ensures only k weights are updated to be non-zero during each round for each client, allowing us to yield the lucrative benefits of sparse gradient computation. Moreover, it ensures each client sends and receives only weights to and from the server at the end and beginning of cach round. Finally, every round of the top-k algorithm, clients ignore the weights not the top-k, potentially causing wasted computation and performance degradation. However, in SPDST, such wasted computations are avoided since only fixed fraction of weights are trained.
E.3 Computation Saving in FLASH
Employing sparse learning in FL helps participating clients roduce communication and compute costs (FLOPs) for training. Without the loss of generality, we now evaluate the convolutional layer training FLOPs for FLASH and demonstrate the relation of parameter density with the reduction in FLOPs and communication cost.
The training FLOPs for layer (Flayer) can be partitioned into forward operation FLOPs (F) backward input (Fl in) and weight gradient (F)ack wi) compute FLOPs. With the assumption of the no-compute cost associated with the zero-valued weights via zero-gating logic (Kundu et 2023b), the Fl lager for FLASH
with parameter density (d <<<<>> 1.0) is
Fl Flayer d x [Fulywd + Fuhack in) + sa x F Fuback wt
(4)
23

---

Published in Transactions on Machine Learning Research (09/2023)
If the zero weights' gradients flow is computed for mask learning, then Fback we can't leverage the advantage of low parameter density. Thus, gradients are dense in JMWST, and Fback wt is the same as that in dense computation. However, for SPDST or JMWST with Tine > 1, zero weights remain zero, allowing us to skip the associated gradient computation safely. This helps extract the benefits of sparsity during all three stages of FLOPs compulation. In other words, FLASH can improve communication and compute costs for clients with limited resources.
E.4 Evaluation of the Communication Cost
Communication cost is associated with transmitting the newly updated weight (or gradients) from client to server or vice versa. The communication reduction achieved by making the weight matrices sparse. In this way, clients do not need to send the whole matrix; instead, they can only send the value of the non-zero ones. In our experiments, communicate them between clients and the server. Finally, communictiion-saving is evaluated by the ratio of full dense model communication cost to sparse model in CSR format.
the compressed sparse row (CSR) format of the sparse model weights to
E.5 Support for Hardware-Friendly Sparsity Patterns
Irregular sparsity is often not well-suited for hardware benefits without dedicated architecture compiler support. However, we can yicld computation energy saving with custom zero-gating logic (Kundu et al. 2020) and compiler support (Liu et al.) 2018). As we intended to achieve reduced computation energy and communication cost FL and as irregular sparsity can yield higher compression than structured/patee patttern sparsity, we have limited our evaluations to random or irregular sparsity only. Nevertheless, we believe our framework can support more complex and structured pruning, which we bricfly explain.
Among the various friindlyy sparsity patterns, the recently proposed N M sparsity [Zouu et al. 2021) has gained significant attention due to its less strict constraints. For SPDST, post Stage 1, sparse mask selection can be easily extended to support the N M sparsity. In particular, for layer 1, instead of random assignment of d' x non-zero mask locations, we can partition the total non-zero elements into G groups, where each group will contain d' x k/ k//² non-zero elements. Here, G is evaluated as k. &//M, representing the total element size out of which we need to have a certain fraction as non-zero, and k represents the total number of weights for that layer. As the masks remain frozen, we maintain the pattern throughout the training for each client to extract the benefit. For JMWST, can adapt this principle in the prune and regrow policy during each client's 's local training. In Table 12 now show results with patlern pruning with N M sparsily for the CIFAR-10 dataset.
Table 12: FLASH with structured sparsity for CIFAR-10 dataset
Data Distribution
Density
Method
Acc (%)
non-IID (a = 1)
0.2
SPDST
86.96
non-IID(a = 0.1)
0.2
SPDST
77.64
24


Reviews:
[{"review_id": "FOsVoldVfh", "reviewer": "Reviewer_W8Q4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1.\tThe author conducted numerous experiments and made several observations, which make their claim about mask consensus promising. The authors proposed a FLASH method that can achieve both computation and communication efficiency in FL by employing sparse learning.\n\n2.\tThe author presents a straightforward takeaway: the smaller the variation of sparse masks during the training process, the better the performance.\n\n3.\tThe extensive evaluation on real image datasets demonstrates the effectiveness of the proposed approach for sparse learning in federated settings.\n\n\nWeaknesses\n\n1.\tThis paper presents interesting new findings, but its technical novelty is incremental. All technique aspects, including mask initialization, SPDST, JMWST, and hetero-FLASH are either based on existing techniques or making minor adjustments.\n\n2.\tThis paper needs to provide a theoretical analysis of the critical observation mentioned in Strengths 2, which lowers the generalizability and reliability of this observation.\n\n3.\tThe paper may also consider discussing the impact of data heterogeneity and mask consensus. Many previous works suggested that data heterogeneity can result in different importance and mask consensus among clients, i.e., personalization. However, the results of this paper indicate that mask consensus should be maintained even under high non-IID degrees.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to improve federated learning sparse training method by introducing sparse training mask consensus to enhance the final accuracy. The authors first observe that the closer the sparse training masks are between training epochs, the better the model converges. Based on this observation, they propose two methods: SPDST and JMWST. SPDST: The server defines a mask before the training starts and keeps it unchanged throughout the training process. JMWST: Each client adjusts their mask locally, and the server uses the aggregated model to sample the new mask, then broadcast it to the clients. Experimental results demonstrate that a high mask consensus is beneficial in model convergence.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The above three weaknesses are critical to the work.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "N2ogH1jZkF", "reviewer": "Reviewer_5Y9b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The empirical result is interesting, especially the success of SPDST which freezes the mask and do client model updating only for the survived links. \n\n**Weaknesses**\n- (Lack of theoretical interpretation) I think it would be great if the authors provide theoretical explanation on the success of the proposed methods, at least for very simple architecture & data setting. Probably focusing on some toy example that NST fails, but FLASH successes is enough.\n\n- (Room for improvement on writing) I think this paper has much room for improvement on writing. I want the authors to clarify the following questions I have\n    - Fig.4 shows Jaccard distance between two models? Which two models are chosen? Global model at round $i$ and round $i+1$?\n    - While explaining Table 2, I could find some weird phrases, e.g., “for the training in row 1”. The terminology “pre-defined w/ mask frozen” is also weird. What is pre-defined?\n        - I couldn’t fully get the schemes in Table 2. Especially, what is the scheme for 2nd row? Do we define the mask based on sensitivity, and then train the survived weights while the mask is frozen?\n        - The author says “pruning sensitivity” is evaluated using another pre-trained model, but how is it trained? I think it is better to provide some visualization of schemes considered in Table 2. \n    - In Table 2 (from row 3 to 5), how do we choose a model that meet the target SM value? We stop training when the average Jaccard distance Fig.4b is less than the target?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focusses on the problem of developing sparse learning algorithms for federated learning (FL). Based on some observation, the authors conjecture that the failure of conventional naive sparse training (NST) method is originated from the lack of consensus in the trained sparsity masks among the clients. \n\nThe authors propose FLASH, a new two-staged framework for applying sparse learning on FL. Once the masks are initialized in the first stage, FLASH apply sensitivity-aware training methods (dubbed as SPDST and JMWST). \n\nExperimental results show that the proposed methods successfully do sparse learning, compared with existing methods. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Add theoretical interpretation suggested above\n- Fix the writing issue listed above\n- Are all the experiments focus on specific local sparse learning (SL) method? If so, adding more experimental results on various local SL methods would be beneficial\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TK4R8qX0vV", "reviewer": "Reviewer_WFQs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is generally well-written. Improving the computation and communication efficiency in federated learning is of potential importance.  \n- The idea behind the proposed FLASH framework is easy to follow.  \n- The experimental results of the FLASH framework (at least for computer vision tasks) are convincing.  \n\nWeaknesses\n- The paper keeps claiming \"computation efficiency\". However, I think FLASH can only introduce random sparsity on local models. For the current generation of hardware, sparse computing is not well supported. Thus, with respect to the actual compute speed, I am not sure if FLASH can attain any speedup.  \n- All experiments are on small-scale vision tasks. It would be great to explore language modeling tasks on larger-scale datasets and models.  \n- Two approaches have been proposed in FLASH (i.e., SPDST and JMWST), however, it is not clear which approach users should use in practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies sparse federated learning, particularly for resource-limited computing devices. The paper observed that directly adopting sparse learning algorithms can cause the lack of consensus of the learned masks, which will further cause the degradation of the learned global model. Based on the observation, the paper proposed the federated lottery aware sparsity hunting (FLASH) framework to achieve both computation and communication efficiency in FL using sparse learning. Experimental results have been shown to demonstrate the effectiveness of the proposed FLASH framework.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Elaborating more on the actual computation efficiency gain of the proposed FLASH framework.\n- Adding larger-scale computer vision (e.g., ImageNet) and natural language processing (e.g., GPT finetuning) experiments.\n- Provide more guidance on selecting between SPDST and JMWST in practice.", "broader_impact_concerns": "I don't find any concerns on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides clear evidence that (1) lack of consensus among sparsity masks in federated learning settings harms convergence an (2) the proposed sparsity masking methods offer clear benefits over related methods in the field. The reviewers agreed that the experiments were quite convincing and thorough in the computer vision setting, and the added experiments in NLP tasks only heighten this.

## Requested changes

Some of the reviewers were concerned with theoretical interpretations of the behavior in this work. I believe that this is unnecessary to provide (TMLR guidelines do not suggest that any paper must have experiments and theory). While I am impressed by the speed with which the authors added a theoretical analysis to the paper, **I would like to request that the authors remove this from the final version**. I believe that this theoretical analysis has not been subject to appropriate peer review, and that it is frankly unnecessary, and possibly a distraction from the strong empirical evidence in the paper.

Reviewers also requested changes related to the organization, clarity, and writing. I would urge the authors to do one last significant pass in improving these, though I wish to emphasize that the authors have **already done a great job of improving the readability of the paper**. I would also urge the authors to ensure that their citation formatting is correct throughout the paper. There is still at least one place where the authors use `\cite` instead of `\citet` or `citep` (the conclusion has a line "compared to SOTA Qiu et al., (2021)" which is difficult to parse).
Decision: Accept (poster)