SparsyFed: Sparse Adaptive Federated Learning
Authors: Adriano Guastella
Lorenzo Sani
Alex Iacob
Alessio Mora
Paolo Bellavista
Nicholas Donald Lane
Venue: ICLR 2025 Poster (2025)
Keywords: dynamic sparse training
federated learning
cross device federated learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Sparse training is often adopted in cross-device federated learning (FL) environments where constrained devices collaboratively train a machine learning model on private data by exchanging pseudo-gradients across heterogeneous networks. Although sparse training methods can reduce communication overhead and computational burden in FL, they are often not used in practice for the following key reasons: (1) data heterogeneity makes it harder for clients to reach consensus on sparse models compared to dense ones, requiring longer training; (2) methods for obtaining sparse masks lack adaptivity to accommodate very heterogeneous data distributions, crucial in cross-device FL; and (3) additional hyperparameters are required, which are notably challenging to tune in FL. This paper presents SparsyFed, a practical federated sparse training method that critically addresses the problems above. Previous works have only solved one or two of these challenges at the expense of introducing new trade-offs, such as clients’ consensus on masks versus sparsity pattern adaptivity. We show that SparsyFed simultaneously (1) can produce 95% sparse models, with negligible degradation in accuracy, while only needing a single hyperparameter, (2) achieves a per-round weight regrowth 200 times smaller than previous methods, and (3) allows the sparse masks to adapt to highly heterogeneous data distributions and outperform all baselines under such conditions.

Published as a conference paper at ICLR 2025
SparsyFed: SPARSE ADAPTIVE FEDERATED TRAINING
Adriano Guastella
Lorenzo Sani
Alex Iacob
Alessio Mora!
Paolo Bellavista!
Nicholas D. Lane
ABSTRACT
Sparse training is often adopted in cross-device federated learning (FL) environ- ments where constrained devices collaboratively train a machine learning model private data by exchanging pscudo- gradients across heterogeneous networks. Although sparse training methods can reduce communication overhead and com- putational burden in FL., they are often not used in practice for the following key reasons: (1) data heterogeneity makes it harder for clients to reach consensus on sparse models compared to dense ones, requiring longer training; (2) methods for obtaining sparse masks lack adaptivity to accommodate very heterogeneous data distributions, crucial in cross-device FL; and (3) additional hyperparameters are required, which are notably challenging to tune in FL. This paper presents SparsyFed, a practical federated sparse training method that critically addresses the problems above. Previous works have only solved one or two of these chal- lenges at the expense of introducing new trade-offs, such as clients' consensus masks versus sparsity pattern adaptivity. We show that SparsyFed simultane- ously (1) can produce 95% sparse models, with negligible degradation in accuracy, while only needing single hyperparameter, (2) achieves per-round weight re- growth 200 times smaller than previous methods, and (3) allows the sparse masks adapt to highly heterogeneous data distributions and outperform all baselines under such conditions.
INTRODUCTION
Federated Learning (MMMahan et al.
2017) has become standard technique for distributed train-
ing on private data Yangg et al. 20188 Ramaswamy et al., 2019 Pati et al. 2022 Wang et al. 2023
Huba et af. 2022 Bonawitz et al. 2019),
particularly on edge devices. Given its application to con-
strained hardware, mitigating communication and computational overhedds signifaad- in standard FL infrastructures (Kirrouz et al. 2021 Bellavista et al.
a key field focus. Practical cross-device FL methods typically assume stateless clients with imbalanced, heterogeneous datasets and constrained, diverse hardware (Wang et al. 2022]. Restricted client hardware and low commu- nication bandwidth significantly increase training time compared centralized methods, elongating hyperparameter tuning (Khodak et al.] (202T) Additionally, unknown data distributions and dynamic client availability demand robust optimization methods that can handle these variations. When de- vice availability is constrained, the federated orchestrator may struggle to sample a representative client subset (Eichner et al.) 2019; Cho al. 2020 Li et al. 2020b), inducing trade-off's between sampling ratio and efficiency (Charles et al.) (0221b.
Sparse training methods improve computational and communication efficiency by reducing (a) memory footprint and FLOPs during training (Raihan & Aamodi 2020). and (b) the communi- cation costs (Bibikar et al. 2022). However, applying these methods in cross-device FL challeng- ing due to client availability and data heterogeneity, which can disrupt the binary mask structure across clients (Qiu et al., 2022). Such inconsistencies hinder consensus on the binary mask, low- ering global model performance (Babakniya et al.) 2023). Recent approaches address these issues using fixed sparse masks (Huang et al. 2022 Quu et al. 2022) or dynamic methods involving mask
Dipartimento di Informatica Scienza e Ingegneria, Università di Bologna Department of Computer Science and Technology, University of Cambridge Flower Labs, UK
Equal Contribution, correspondence to tuaiiogugggtt@@uuuieaaa

---

Published as a conference paper at ICLR 2025
warmuu and refreshing (Babakniya et al. 2023). However, fixed masks reduce the adaptability to unseen distributions, while dynamic methods require careful tuning of additional hyperparameters, like warmup duration and refresh interval. Fixed-mask methods also limit model plasticity- the ability to rewire and adapt to diverse distributions (Lyle et al. 2022 2023). For example, it is known that in multi-task or continual learning setting, neural networks can be iteratively pruned to build task-specific sub-networks (Mallya & Lazebnik] 22018). a lost ability when adopting a fixed-mask method. This lack of adaptability makes fixed-mask approaches unsuitable for cross-device FL, where unseen distributions frequently arise. Thus, we argue that sparse training methods for cross- device FL should (a) adopt dynamic masking and (b) remain agnostic to optimization and selection methods while minimizing hyperparameter complexity.
Sparse Training
Activation Pruning
Weag by
909 B5 800
pan-ager apaniyy ln 4, TE
① M
SAR ATM
Server
Spane
00000 doo
Client1
Client
Client
Gradient
Gondrrt
Figure 1: SparsyFed pipeline. (1) Server broadcasts the global model We- (2) Client re- parameterizes local weights. (3) Executes a forward pass on batch B. (4a) Computes layer- spar- sity s, (4b) Prunes activations using s, and stores them. (5) Computes grads. (6) Applies grads. (7) Computes model updates and applies Top-K pruning. (8) Sends sparse updates Auح back to the server. (9) Apply server optimizer to obtain the global model. Steps (2-6) repeat until convergence.
Our work addresses these challenges by introducing SparsyFed, a sparse training method for train- ing global models in cross-device federated learning. During local training, as illustrated in Fig. T SparsyFed uses an easy-to-tund approach to prune (1) activations with an adaptive layer-wise method and (2) model weights before communication using the target sparsity. These strategies can reduce FLOPs and memory usage, with suitable hardware support (Raihan & Aamodi 2020), and the com- munication costs by transmitting sparse updates. Our layer-wise approach prunes based on each layer's parameter proportion, unlike previous methods (Babakniya et al., 2023] Qiu et al. 2022), which applied fixed global sparsity across layers. This approach removes more capacity from dense layers while preserving parameter-efficiee round, allowing the complete flexibility of the local optimization. SparsyFed provides robust initial- ization for crucial layers, such as embeddings, without excluding layers to preserve performance as done in past works (Qiu et al., 2022 Raihan & Aamodt| 2020). Additionally, SparsyFed employs a sparsity-inducing weight re- pa-aaaeerriiztiion (Schwarz et al. 202T) based on a single parameter, enhancing the model's resilience to sparsity and improving adaptivity, which enables changing the global mask for new clients with diverse data distributions. SparsyFed is agnostic to the choice of the outer optimizer, treating sparse model updates as pseudo-gradients. It is also compatible with biased client selection policies, allowing training masks to adapt to the training data utilized. Our work's contributions are:
ones. Sparsyfed also prunes models at the end of each
1. We introduce SparsyFed, a method which accelerates on-device training in FL. SparsyFed achieves high sparsity (up to 95%) without sacrificing accuracy through a novel approach combining hyperparameter-fiee free activation pruning and weight re- re-paramtterization.
2. We compare SparsyFed against the latest state-of-the-ee techniques, demonstrating supe- rior accuracy over sparse training baselines (including those using fixed sparse masks) and, in some cases, surpassing non-pruned baselines at extreme sparsity levels.
3. SparsyFed quickly achieves consensus client sparse model masks, enabling faster global convergence and significantly reducing downlink communication costs (up to 19.29 x).
4. We evaluate SparsyFed with ablation studies typical cross-device FL datasets, including CIFAR-10/100 (Krizhevsky) (012) and Speech Commands (Warden 2018), under various data heterogeneity conditions.
We provide the developed code publicly available in this repository to facilitate result reproducibility and for the community of researchers in the field.
2

---

Published as a conference paper at ICLR 2025
2
BACKGROUND
In the following, describe typical sparse training and weight re-parametrization techniques, which are key components of our work, and discuss their relevance to cross-device FL.
2.1 CROSS-DEVICE FEDERATED LEARNING
Cross-device FL (Kairouz et al. 2021 involves the distributed training of a machine learning (ML) model across a population of edge devices exchanging model updates with central server through heterogeneous networks (McMahan et al. 2017 Li et al., 2020a). Clients in these settings usually possess minimal data samples and very heterogeneous data distributions Srabutiios(Kairuuz Kairouz et al. (202I). Given the constraints of edge devices (e.g.. limited processing power and battery life) and the heterogeneity of networks (e.g.. diverse bandwidth), computational- and communication-efficiii
training methods
crucial for practical FL (Bonawitz et al. (209)). The research community has proposed means optimize FL. for communication efficieccy (Sattler et al. 2019 [Jiang
Jiing et al. (003)) and computational
efficiency (Horvath al. 2021 Niu al. 2022 Mei al. 2022). Our approach aims to optimize both by leveraging sparse training.
2.2 SPARSE TRAINING
In centralized settings, sparse training tries to learn sparse model during training to achieve model compression, lower computational demands during training, or faster inference. typical sparse training pipeline tends to start with random sparse network and follow a cycle of regular training, pruning, and regrowth (Mocanu et al., 2018; Mostafa Wang 2019 Dettmers Zeitlemoyer, 2019 Liu al. 2020). acting only model parameters. A more advanced technique relevant to our work, Sparse Weight Activation Training (SWAT) (Raihan & Aamodt) 2020), tailors the forward and backward passes, acting both activations and model parameters, to induce sparse weight topology and reduce the computational burden. In each forward iteration, SWAT selects the Top-K weights based magnitude, using only these the active weights. Only the highly activated neurons associated with Top-K K. pruned activations are considered for backpropagation during the backward pass. Notably, full gradients are still applied in the backward pass, allowing updates to both active and inactive weights. This mechanism enables the dynamic exploration of different network topologies throughout the training process.
2.3 WEIGHT RE-PARAMETRIIATTIO
Weight re- parametrization (alimans & Kingma 2016; Li al. 2019 Gunasekar et al. 2017 Miy-
ato al. 2018 Vaskevicius al. 2019 Kusupati et al. 2020 Schwarz et al. 2021 Zhao et al. 2022) in neural network training involves restructuring how weights are represented to improve training dynamies, optimize convergence, introduce specific properties such sparsity, without changing the network's 's architecture. Particularly relevant to our work, [Schwarz et al. (202T) pro- pose sparsity-inducing weight pe-arammtrrizaiion that aims to shift the weight distribution towards higher density near zero, aiding in pruning low-magnitude weights. This simplification emerges by raising the model parameters to the power of 1 during the forward pass while preserving their sign. The re-parametrized weight vector component w is computed as w sign(v) |u] where v represents the original weight vector component, and small-valued parameters receive smaller gradient updates, while large-valued parameters receive more significant updates, reinforeing "rich get richer" dynamic.
scalar value. Due the chain rule,
SPARSE ADAPTIVE FEDERATED TRAINING
In the following, we present our SparsyFed method for sparse training in cross-device FL settings. Our method introduces a novel approach based on activation pruming and weight parametrization, applicable to any cross-device FL setting for obtaining sparse global model. SparsyFed reduces the computational and communication overhead of highly heterogeneous FL environments, adapting the sparse mask of the global model. The procedure is outlined in Algorithms Tnd
3

---

Published as a conference paper at ICLR 2025
Algorithm Sparse federated training pipeline of SparsyFed.
Require: wig initial model's weights, 3: weight re- parametrization exponent, 8: target sparsity
Require: T: number of federated rounds, E: number of client local epochs per round
Require: P: clients population, π) = ntt): learning rate scheduler as function of round Ė
Require: (D: clients' datasets, B: local batch size
1: procedure SparsyFed
2: for (= O..... - do
3:
Server samples a subset of clients S; - for each client E St in parallel do
4:
5:
Wi,0 <- Wt
6:
for 1 = 0, .....
do Wi,++1 + Sparseclientopt(urn D,, B,)
7:
> See Alg Compute pseudo-gradient Adg f Top-K (Aw,, 8) & Prune Awf w/ global unstructured Top-K using target
8:
Aw -- Wi,E we
9:
10:
Wt+1 OuterOrI(ww) (A0 hes.) Server optimization, e.g., Reddi et al. (2021)
11: return WT
Assumptions on the FL setting. As in any cross-device FL setting, the training is orchestrated by parameter server (McMahan et al. 2017) that in charge of initializing the global model, sam- pling a subset of clients every Tederated round, aggregating the pseudo-gradients after clients have trained on their local datasets. By following practical considerations (as extensively discussed in Bonawitz et al. (2019); Wang et al. (202T)), SparsyFed does not require any particular assumption on the client selection policy, nor on the server optimizer, nor the client optimizer. Thus, our algo- rithm's 's design allows it to benefit from any present or future federated optimizer practitioners, and researchers may develop without losing its properties. In particular, SparsyFed only requires addition of one hyperparameter compared to standard dense training, whose sensitivity is discussed in Appendix E.2, making it most suitable for cross-device FL where the hyperparameter optimiza- tion (HPO) is challenging (Khodak et al. 2021). We also present hyperparameter free alternative in Appendix E.2.2 After initialization, the parameter server iteratively samples clients, broadcasts from clients, and aggregates
the latest version of the global model, collects the pseudo-gradied the updates for obtaining the new global model, as described in Algorithm [ As such, we assume agnosticism w.r.t. the server optimizer (line 10 in Algorithm I to allow practitioners use their preferred one, e.g., ServerOpt in Reddi et al. (2021).
Algorithm 2 Sparse Client Optimization of SparsyFed
Require: wny initial model's weights, 3: weight re- parametriizaioon exponent, 7y: learning rate
Require: D (x,, ii....... dataset composed of N samples with inputs x; and outputs y4
Require: B: batch of data samples, B: batch size, T = N number of batches, F: cost function
Require: =(.,)): function computing the sparsity of the layer 0., at time
Require: size(f): computes the number of layers in the model /
Require: GetLayer(a,.)) (): function extracts the weights and activations for the current layer.
Require: SetLayer ()) function updates the sparse weights and activations in the model.
1: procedure SPARSECLIENTOPTOONn 3,
2:
for each step
do
3:
B, Wtt.Neettiinnnaattcccop.c.
4:
0. < sign(w; | 8
Point-wise weights re-parametrization (Schwarz et al. 2021 Forward pass compute activations
5:
0t + Forward B.) for each layer 0, o,..., 0i...,,ize(())) do
6:
7:
0,,, 01,1 t (e, 1), GetLayez(a..))
8:
s(Or.)
9:
âr t K(op-K(a,,,sL.)
Prune az w/ unstructured Top-K using target $1,1
10:
å, t SetLayer((n..))
9: < F (.)
D Compute gradients using sparse activations
12:
W/+1 = cl clientOpt 1,
D Apply client optimizer, e.g., Reddi et al. (2021)
return WT
4

---

Published as a conference paper at ICLR 2025
Sparsity-Inducing Weights Re-parametrization. Before the local forward pass at step t, we apply a sparsity- re-parametrizatinn to the local model weights wt. producing 02 (line 4, Algo- rithm 2). As discussed in Section 2.3 re-parametrization techniques have been widely studied for various purposes. In our case, we aim to enhance pruning effectiveness and efficiency by inducing sparsity directly through optimization and adapting input data. These outcomes are particularly beneficial in cross-device FL settings, where datasets are highly imbalanced, and some clients may only have a few samples (Section 2.1 From the available sparsity-inducing methods, we adopt Pow- erpropagation (Schwarz et al. (022)) due to its simplicity (introducing only one hyperparameter) and its ability to preserve the neural network's functional relationships during training. Furthermore, avoids introducing non-uniform biases across layers, which improves its compatibilityy with pruning. Every parameter weight w E w is transformed into v = sign(us) |we| f where sign(w) is the sign of w, |w]] is its LI norm, and ,3 is the Powerpropagation parameter. This pe-arammtrrizaioon enhances global training by promoting client consensus since the dynamics introduced naturally guide train- ing toward the subset of non-zero weights in the model. Weights transitioning from zero to non-zero during training typically have smaller magnitudes, limiting their impact on updates. This ensures that clients focus shared subset of weights, facilitating aggregation.
Pruning Activations During Local Training. The local training procedure of SparsyFed is out- lined in Algorithm P and relies on two main pillars. First, it ensures that the edge devices benefit from the model's sparsity by reducing memory footprint and FLOPPs (depending on the underlying implementation, see Appendix H.. Second, it guarantees the retention of as much information as possible during the training. We follow Raihan & Aamodt (2020) in three aspects. First, we use dense activation vectors (a,, line 5) retain all learned information during the forward pass. Sec- ond, we prune activations before the backward pass by aligning their sparsity with the weight vectors (lines 7-10). Specifically, activations are pruned layer-wise wise using the Top- method with target per-layer sparsity level (st,2, line 8), determined by the corresponding per-layer weight sparsity (line 9). Such pruning involves retrieving the weights and activations per layer (line 7) and updating the pruned activations before computing gradients. The weight parametrization preserves high sparsity throughout training (Fig. [2] in the appendix), ensuring consistent patterns between weights and ac- tivations and seamless integration into the model's sparse structure. Third, we keep gradient vectors (gt. line 11) dense to avoid losing crucial information during updates. Pruning activations before the backward pass reduces computational cost while maintaining dense gradients for robust updates. Notably, initial model weights remain unpruned to allow meaningful training initialization, as clients train a dense model in the first round. With these principles, the local training procedure is designed to be optimizer- aptimizer-amosticc, e.g., compatible with ClientOpt (Reddi et al. 2021).
Pruning model parameters before communication. The data-driven and hyperparameter-less pruning procedure described above requires further step to ensure compliance with the communi- cation requirements. Clients receive a model parameters target sparsity, & from the server, which must be met before communicating the pseudo-gradien updates. Thus, the client applies global unstructured pruning step based on Top-K K using the target value 5 for the output sparsity, guaran- teeing to save communication costs using a single parameter. Notably, this allows for non-uniform sparsity across layers, which has been proved to help maintain performance (Kusupati et al. 2020).
EXPERIMENTAL DESIGN
Datasets and tasks. We selected three datasets assess SparsyFed'ss performance: CIFAR-10/100 (Krizhessky| 2012), and Speech Commands (Warden 2018). CIFAR-10 and CIFAR- datasets contain 32 x 32 color images of 10 and 100 classes, respectively. The Speech Commands dataset includes audio samples of 35 predefined spoken words and is used for a speech recognition task. Since these datasets' samples can be easily distributed across a pre-defined number of clients, they common for simulating the heterogeneous data distributions of federated learning settings. On all datasets, we trained models for multi-label classification tasks.
Data partitioning and sampling. The datasets above are distributed among 100 clients and parti- tioned using the method in Hsu et al. (2019), simulating various degrees of data heterogeneity. The distribution of labels across clients is controlled via concentration parameter that rules Latent Dirichlet Allocation (LDA), where a low a value translates to non-IID distribution and high value to the IID distribution of labels. Specifically, we refer data distributions as III for 10 and
5

---

Published as a conference paper at ICLR 2025
non-IlI for 1.0 and a 0.1. To ensure reproducibility, we fixed the seed to 1337 for the LDA partitioning process. The federated orchestrator randomly sampled 10 clients out of the 100 clients in the population every round.
Model and training implementation. We employed a ResNet-18 (He et al. 2016) backbone for all experiments, adapting the classification layer each specific task depending the number of classes. ResNet-18 was chosen for its size, its popularity in the area, and the scalability of the ResNet family. While our training pipeline is implemented with PyTorch (Paszke et al. (019). we designed custom layers and functions for some of SparsyFed's components, such as layer-wise activation pruning and weight parametrization. We used F'lower (Beutel al.) 2022) to simulate the federated learning setting. All models were trained from scratch without relying on any pre- trained weights.
Sparsity ratios. In our experiments, we targeted different values for the sparsity ratio in the set (0.9, 0.95, 0.99, 0.995, 0.999). We chose 0.9 as the minimum value because it has been shown to bring effective gains (Frankle & Carbin 2019) for both memory footprints and FLOPs (Ap- pendix H). Our investigation spans to the extreme value of 0.999 to fairly present the downsides of these sparsity ratios. We applied the same target sparsity for all devices in the federation. For completeness, we show in Appendix E.6 how SparsyFed performs when adopting heterogeneous sparsity targets among devices in the federation.
Communication costs. Measuring the communication costs of SparsyFed crucial for understand- ing its practical implications cross-device FL settings. To make analysis agnostic to any compression technique implementation for sparse unstructured models, we report the costs as the number of non-zero parameters effectively exchanged during the FL. Therefore, the communication cost is derived from the effective sparsity of the transmitted model, considering both the downlink (server-to-clients and uplink seients-to-server communication steps. For clarity, we calculate the communication cost as if only one client participated, assuming all communication occurs in par- allel. This ensures that our measurements reflect the total communication load without temporal delays between clients and avoids any bias introduced by the varying sampling proportion of clients in each round.
Reproducibility. Three different seeds were used for client sampling (5378, 9421, and 2035), while other stochastic processes were seeded with 1337 for reproducibility purposes.
EVALUATION
This section discusses the evaluation of SparsyFed for adaptive sparse training in FL cross-device. The experimental results shown here aim to answer the following research questions.
Can SparsyFed mitigate the expected accuracy degradation at high and very high sparsity levels? We compare against the baselines, i.e.. Top-X. K, ZeroFL (Qiu et al. (022), and FLASH (Babakniya et al. 2002)), for both the accuracy and the communication costs (Sec-
tions 5.1 and 5.2 respectively).
2. How does our adaptive sparsity pattern interact with the heterogeneous data distributions compared to other sparse training methods? Similarly to BBaakkniya et al. 2023). we ana-
lyze the consensus across clients on the sparse pattern (Section (53).
3. How do the main components of SparsyFed contribute to maintaining the accuracy dif- ferent sparsity levels? We ablate the activation pruning step (Section 5.5) and vary the weight re- paraetttttriiiioonnn (Section 5.4) answer this question.
To ensure a fair and comprehensive comparison, we reimplemented state-of-the-art methods in our experiments, including ZeroFL, FLASH, and Top-K pruning. A detailed description of these base- line methodologies and their comparisons against SparsyFed is presented in Appendix G
5.1 ACCURACY DEGRADATION
The analysis discussed in this paragraph demonstrates SparsyFed's resilience to the performance degradation typically associated with pruning. as SparsyFed achieves competitive results even at high sparsity levels. Compared to other competitive methods under the same sparsity constraints, TTbll that SparsyFed exhibits the lowest accuracy drop across all settings and target sparsity

---

Published as a conference paper at ICLR 2025
Datavet
Spuniity
RosNet-18
ZeroFl.
FLASH
Sparty
R8sNet-18
ZeroFl
FLASH
SpanyyFal
dense
83.70
1.70
73.81 + 4.84
0.9
80.56 = 1.00
76.16 + 1.30
81.15
+ 1.03
82.13
1.53
69.79 + 3.78
67:40-44
4-111
71.87 + 2.63
75.00
2.78
CIRAR-10
0.95
74.71 3.29
27
75.53 1: 2.27
79.36
1: 1.03
1.37
82.00
77.71
1.58
43.96
60.00
1 4,66
11.99
61.55
51.71
1.4 4.18
3.54
72.08
56.91
1: 2.09
3.55
63.69
75.95
3,90
3.39
0.995
56.02
3,95
1.60
70.01
0.43
19,02
10:77
3.87
56.79
0.999
31.79 + 19:10
17.66
+ 8.34
36,07
+ 7.49
$1.39
3,19
11.50 + 4.49
18.76
4:28
29.31 + 6.75
43.68
7.61
dense
52.29 1.14
48.34 1: 2.71
46.57
1.71
40.70 1: 4.72
51,99
0.21
53,88
0.90
41,96 1: 2.16
31.92 7.65
45.39 4: 0.75
48.37
1.73
CIPAR-100
0.95
19.65
23.27
18.97
2.08
1.75
4.08
1.88
52.81
1.72
17:51
34.21 7.65
13:00
44.31 +
3.38
48.27
41,003 2.14
0.995
9.51 14.81
6.01
+ 4.74
36.43
4.97
42.21 :: 1.03
0.14 + 0.72
7.04 = 5:25
26.44
17.35
35.72
2.01
0.999
3.81 + 2.18
1.96 1: 0.66
5.80 1: 2.86
15.96
0.64
0.14 à: 0.72
1,66 A: 0.97
3.56 d: 2.07
13.84 1: 3.69
dense
91 49 0.94
80.15 +: 2.69
28
87.79
1.40
88
1:72
92.32 1.59
0.97
70.35
77:15 + 0.77
79.67
Speech
78.58
84.29
1.50
0.49
1.04
65.90
71.28 1.35
75.46
Commands
0.99
65.01 0.84
57.79
+: 0.82
09:22
1.50
75.82
3,72
50.42
2: 6.26
41.42
1.60
53.55 4: 2.00
56.69
4.56
0.995
56.73 1.00
37.16
1 2.71
58.23
1.84
68.02
3.14
34,20 1 1.43
22.61
4 3.45
43.16
4. 3.47
48.30
5.39
0.999
21.56 + 12.79
10,10 + 4.01
17.70
2.58
47.43
1.66
19.25 + 6.01
8,85 + 3.76
17.14 + 2.97
29.24
2.34
Table 1: Aggregated results for CIFAR-10, CIFAR-100, and Speech Command datasets, with
ResNet-18, 18, ZeroFL, FLASH, and SparsyFed implementations.
levels. A noticeable drop in performance compared to the dense model was observed only at 99% sparsity. This advantage arises from using weight re-parameterizaaton, which, in some cases, can even enhance the performance of the dense model. Additionally, the minimal performance drop at lower sparsity levels (90- 95%) allows SparsyFed to outperform the dense model in specific scenarios. To stress more method capabilities, (99.9%) where SparsyFed is no longer able to retain sufficient accuracy. Our results show that all the baselines struggle train effectively under such conditions. The adaptivity of SparsyFed's s sparsity patterns promotes consistent performance across clients, even in highly sparse settings, leading to a more synchronized and globally pruned model.
increased the target sparsity to the point
5.2 COMMUNICATION COSTS
The promising accuracy achieved at very high sparsity ratios makes SparsyFed particularly suit- able for cross-device FL settings, where communication costs are a critical concern. As illustrated in Fig. 2 (left), SparsyFed significantly outperforms the baselines regarding both communication savings (19.29x less communication costs compared to the dense model and 1.66x compared to ZeroFL) and preserving required accuracy (consistently above 45%). FLASH has comparable communications costs, 0.97x x ours, but results in lower accuracy. Notably, SparsyFed consistently achieves higher accuracy for given communication cost than the baselines, as shown in Fig. 2 (left). This advantage arises from SparsyFed's s ability to prune weights during client training and maintain close-to-target sparsity ratio during server aggregation, effectively reducing uplink and downlink communication costs. Importantly, FLASH does not see an increase in model density af- ter aggregation due to its fixed-mask local training regime, which prevents weight regrowth. This characteristic results in stable communication costs. In contrast, ZeroFL experiences a substan- tial increase in model density after aggregation, leading to a systematic and significant increase in downlink communication costs.
5.3 CONSENSUS ON THE SPARSE MASKS ACROSS CLIENTS
Achieving consensus on the sparse masks of the clients' updates after each training round is cru- cial for achieving high accuracy. This consensus dynamic can be interpreted as the global model stabilizing around the target sparsity level as federated rounds progress. As shown in Fig, 2 (right), SparsyFed demonstrates minimal deviation from the target sparsity (90%). whereas ZeroFL and Top-K drop below 47% and 83%, respectively, during the initial training stage. SparsyFed's s con- sistency allows clients to effectively collaborate training the same subset of parameters, which reduces the need for excessive pruning after local training and helps retain more helpful informa- tion. Such consistency appears despite our clients being allowed to dynamically change their mask during training, unlike ixed-mask approaches. For all methods, local updates maintain consistent sparsity due to post- toast-rraiingg pruning; therefore, any increase in the global model's 's density indicates that local weight regrowth during training alters the local masks. Once these altered updates ag-
7

---

Published as a conference paper at ICLR 2025
ZenoFI
1est Accurncy (%6
300
Tip-
0.3
Dirnen FLASI
Sparsity (96)
80
70
0.2
ZenFL
0.1
Spuciyeo
Top-R
0.0
10'
10P 10"
500
700
Communication
Figure 2: (left) The plot on the left compares accuracy versus communication cost for four imple- mentations: ZeroFL, Top-K, FLASH, and SparsyFed, with the dense model as a reference. The test is conducted on CIFAR- 100 partitioned with LDA(a 0.1) and 95% sparsity. SparsyFed out- performs the baselines, achieving high accuracy and communicating less. (right) The plot on the right shows the global model sparsity level, measured on the server after aggregating local updates (CIFAR-100,
0.1). The density gain reflects mismatches between client updates, causing the aggregated model to regain density, which can degrade performance and increase downlink commu- nication. Note: FLASH maintains target sparsity after the first round with a fixed mask.
Sparyyfc
To-
ZonfL
Nomd
Figure 3: Intersection over Union (loU) of global model binary masks between training rounds for SparsyFed, Top-K, and ZeroFL (CIFAR- = 0.1, 95% target sparsity). The IoU is calculated between each mask and all other masks across rounds to show changes over time. The x and axes represent training rounds indices-the diagonal indicates the identity. Higher IoU values (close to 1.0) signify stronger similarity between masks, while lower values indicate significant changes. SparsyFed shows consistent mask movement with minimal variation, suggesting strong consensus on weight usage among clients. ZeroFL struggles to find mask consensus, with masks continuing to shift even in later rounds. Note: FLASH is absent since the global mask is fixed.
gregated, the result is denser global model on the server. This misalignment among clients' masks can negatively impact both communication efficiency and accuracy. Global training appears to ben- efit from this consensus in terms of convergence, as shown in Fig. 2 (left), where both SparsyFed and FLASH outperform Top- K despite similar overall communication costs.
5.3. SPARSITY PATTERN DYNAMICS
We analyze the global model's sparse mask dynamics during training to shed light on SparsyFed's ability to maintain stable sparse pattern driven by two key factors: (1) focused weight utiliza- tion and (2) sparse mask adaptivity. First, the weight re-parameterization method targets a critical subset of weights, concentrating information where
most impactful. This targeted approach improves training efficiency, enhances client collaboration, and reduces communication costs, as shown in Fig. 2 Second, unlike fixed-mask methods like FLASH, SparsyFed promotes dynamic mask adaptation, achieving robustness to heterogeneous data distributions throughout the training process. This allows a natural warm-up phase during which the sparsity pattern emerges organically, shown in Fig. 3 (left). Clients rapidly converge
stable shared mask, ensuring consistent per- formance. In contrast, on Fig. (center), Top-K never fully settles on a stable mask, resulting in continuous variation across rounds while being faster in its initial rounds. In Fig. 3 (right), ZeroFl.,
8

---

Published as a conference paper at ICLR 2025
though more robust initially, struggles to maintain performance as it forces weight regrowth to adjust the mask, leading to instability in later rounds.
We do not consider FLASH in this comparison since it fixes its mask after the first round, clim- inating any changes in the sparse mask pattern but reducing mask adaptivity entirely. While this strategy ensures that the model adheres Specifically, since the mask is determined based on the data distribution observed in the first round, FLASH becomes highly dependent the initial client data. This lack of adaptivity contrasts with our method, which allows continuous changes. Thus, it is more effective at handling shifts in data distribution across rounds.
fixed sparse structure, it introduces potential drawbacks.
5.4 ABLATION ON WEIGHT RE- PARAMETERIZATION
In this ablation study, we evaluated various weight p--arameterrzation techniques for their ability to sustain sparsity-driven model while preserving dense-like performance. Since the sparse activa- tions during the backward pass rely on a sparse weight model, the model must maintain high spar- sity levels throughout training while minimizing accuracy loss.
90
Test Aceuacy (%)
70-
Re (Dense)
Tap-K(nn (we (-paramerrinivon))
We evaluated three approaches: fixed-mask train- ing, spectral re-parameterization (Miyato et al. 2018). and Powerpropagation (Schwarz et al. 2021. We provide more details on the spectral re-parameterization in Appendix D.3 Each tech- nique was applied to a ResNet-18 model trained with sparse activations for the backward pass and Top-K unstructured pruning at the end of each training round. This setup enabled direct com- parison with baseline model that lacked any re- parameterization. Among the methods, Power- propagation proved to be most effective for our use case (Fig. 4. demonstrating superior resilience to preserve the accuracy with minimal degrada- tion. By leveraging a "rich get richer" dynamic, Powerpropagation naturally induces sparsity dur- ing training. This enables the model to remain sparse during training, complementing our sparse activation backward pass and improving the model's 's overall efficiency and effectiveness.
Spectril
Flaod Mask
Powerpoopaatiin
10
90
95
99 98.9
Sparsity Level (%)
Figure 4: We report the test accuracy of dif- ferent re-parameterization methods with sparse activations during backpropagation. We de- ployed ResNet-18 18 trained on the CIFAR-10 dataset using LDA(a 1). This plot illus- trates the methods' performance under differ- ent sparsity levels. Powerpropagation exhib- ited superior robustness to the applied sparsity levels, achieving the best overall performance among these methods.
5.5 ABLATION ON ACTIVATION PRUNING
The activation pruning step in SparsyFed (lines 6-10 in Algorithm 2 designed to reduce compu- tational costs of local training, which is particularly sensitive when executing on edge devices. This decision is motivated by prior studies Section 2 and Appendix H As part of our ablation studies, we analyzed the impact of this pruning step on the accuracy and overall performance of the model. Since pruning activations during the backward pass do not directly affect weight density, we focus explicitly on its influence test accuracy.
In Table 2 we compare SparsyFed with and without activation pruning during the backward pass. Our results show that activation pruning minimally impacts test accuracy at higher density levels. Significant performance degradation occurred only under extreme sparsity, where excessive pruning in specific layers substantially reduced activations, leading diminished overall performance. UI- timately, the computational speedup achieved through activation pruning validates its inclusion in SparsyFed, as it balances efficiency with model accuracy.

---

Published as a conference paper at ICLR 2025
a 10 (IID)
1.0 (non-lID)
a 0.1 (non-lID)
Sparsity
no Act. Pr.
Act. Pr.
no Act. Pr.
Act. Pr.
no Act. Pr.
Act. Pr.
0.90
83.92 + 1,58
84.31 + 0.86
82.27 + 2.21
82.13 + 1.23
76.60 = 1.54
75.00 = 2.78
0.95
83.80 + 0.90
84.25 + 1.38
82.6 = 1.58
75.29 = 2.64
75.95 = 3.39
0.99
77.54 1.98
77.16 0.85
75.76
77.71 1.69
63.79 3.96
63.69 = 3.90
0.995
72.71 0.65
70.89 + 2.22
70.01 1001=:0: 0.43
59.15
56.79 = 3,97
0.999
62.12 +: 1.74
55.24 + 2.09
62.67 + 2.19
51.39 = 3.191
49.43 = 2.45
43.68 = 7.61
Table 2: Accuracy comparison between SparsyFed with and without the pruning of the activation, on CIFAR-10 with LDA 10",
1.0, and = 0.1.
RELATED WORK
Sparse training in centralized settings. Methods to enforee sparsity in neural networks can be grouped into two main categories: (1) dense-to-sparee sparse methods that train a dense model and achieves a sparse model after training (Molchanov
al. 2017 Louizos et al.) (007). (2) sparse-to-
methods where pruning happens during training (Louizos et al. 2018 Dettmers & Zettlemoyer 2 et al. Jayakumar et al. [02T] computational requirements (Bengio et al. | 2015) and speeding up training. Our work introduces
| Raihan & Aamodt 2020). thus theoretically reducing
novel method to implement sparse training directly within FL clients, inspired by sparse-to-sparse approaches in centralized settings such as Raihan & Aamodt (2020).
Pruning model updates in FL. sraen-trannin pruning, where clients first train dense models and then prune updates, is a common approach in FL (Sattler et al.) [2019] Wu et al.) 2020 Malekijoo eet al. 202T). Sparse Ternary Compression (Sattler et al.) (0019) combines Top- K pruning with ternary quantization to compress client updates, while FedZip (Malekijoo et al., 2021) uses layer- wise pruning, and FedSCR (Wu et al. 2020) employs patterns in client updates for more aggressive compression. However, these methods primarily improve communication efficiency without reduc- ing computational overhead, as they still train dense models. In contrast, our method trains sparse models from the start, resulting in sparse updates and maintaining sparsity even after server aggre- gation, ensuring efficient upstream and downstream communication.
Sparse training in FL. Several studies have explored sparse learning in federated settings (Bibikar et al. 2022 Huang et al. 2022 Jiang et al.) 2023 Qiu et al. 2022 Babakniya et al. (202)), but each has limitations. FedDST Bibikar et al. (022) applies Rigl. (Evei et al. 2020) to train sparse models, focusing on heterogeneous data without addressing extreme sparsity levels. FedSpa (Huang et al.) 2022) uses fixed sparse mask throughout training without a clear rationale behind it. PruneFL (iang et al. 2023) computes sparse mask using biased client data and requires full gradient uploads, increasing communication costs. ZeroFL (Qiu et al. 2022) integrates SWAT (Raihan & Aamodi 2020) for local training but struggles with weight regrowth, requiring pruning after each round, which can cause information loss. FLAASH (Babakniya et al. (002)) introduces a fixed mask after warm-up phase, but it depends heavily the chosen clients during the first sampling and does not adapt to concept drift. In contrast, our method uses a dynamic sparse mask, offering more flexibility and better performance highly non-IID FL settings.
7 CONCLUSIONS
This work presents SparsyFed, an adaptive sparse training method tailored for cross-device federated learning (FL.). We show that SparsyFed can achieve impressive sparsity levels while minimizing the accuracy drop due to the compression. SparsyFed outperforms in accuracy three federated sparse training baselines, Top- K, ZeroFL, and FLASH, using adaptive and fixed sparsity for three typical datasets used in cross-device FL. We were able to ensure a limited drop in accuracy at sparsity levels of up to 95%, achieving up to a 19.29x reduction in communication costs compared to dense baselines. The results presented in this work make our proposal particularly suitable for cross-de device FL settings, which may require extreme communication cost reductions and the capability to adapt to heterogeneous distributions across federated rounds.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was supported by the following entities: The Royal Academy of Engineering via DANTE (a RAEng Chair); the European Research Council, specifically the REDIAL project; Google through a Google Academic Research Award; in addition both IMEC and the Ministry of Education of Romania (through the Credit and Scholarship Agency); and partly supported by the the Future" (PE00000001 pprogram "RESTART") and by the National PRIN JOULE project. Furthermore, we thank Javier Merraesez-Marques, Stefanos Laskaridis, Xinchi Qiu, and Francesco Corti for their invaluable feed- back and collaborative spirit throughout this research project and the ICLR reviewers for helping improve our paper.
European Union under the NRRP partnership on "Télecommunications
REFERENCES
Sara Babakniya, Souvik Kundu, Saurav Prakash, Yue Niu, and Salman Avestimehr. Revisiting sparsity hunting in federated learning: Why does sparsity consensus matter? Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL htl/:ttt://pppeeeveee..ne
Forum?id=iRyhppsp
Paolo Bellavista, Luca Foschini, and Alessio Mora. Decentralised Learning in Federated Deploy- ment Environments: A System-Level Survey. ACM Computing Surveys (CSUR), 54(1):1-38,
2021.
Emmanuel Bengio, Pierre-Lue Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint
2015.
Daniel J. Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Kwing Hei Li, Titouan Parcollet, Pedro Porto Buarque de Gusmão, and Nicholas D. Lane. Flower: A friendly federated learning research framework, 2022.
Sameer Bibikar, Haris Vikalo, Zhangyang Wang, and Xiaohan Chen. Federated dynamic sparse training: Computing less, communicating less, yet learning better. In Thirty-Sixth AAAI Confer- ence on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artifi- cial Intelligence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pp. 6080-6088. AAAI Press, 2022. doi: 10.1609/AAAI V3616.22555.
Kallista A. Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Konečný, Stefano Mazzoechi, Brendan McMahan, Ti- Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: System design. In Ameet Talwalkar, Virginia Smith, and Matei Zaharia (eds.), Proceedings of the Second Conference on Machine Learning and Systems, SysML 2019, Stanford, CA, USA, March 31 April 2, 2019. mlsys.org. 2019.
Zachary Charles, Zachary Garrett, Zhouyuan Huo, Sergei Shmulyian, and Virginia Smith. On large- cohort training for federated learning, 2021a.
Zachary Charles, Zachary Garrett, Zhouyuan Huo, Sergei Shmulyian, and Virginia Smith. On large- cohort training for federated learning. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 20461-20475, 2021b.
Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. CORR, abs20000001233, 2020.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.1

---

Published as a conference paper at ICLR 2025
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale, 2021. URL https: //axiiv///ivv.v/aa wwg/aag//ga....ee 2010 1.001.100 11929
Hubert Eichner, Tomer Koren, Brendan McMahan, Nathan Srebro, and Kunal Talwar. Semi-cyelic stochastic gradient descent. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceed- ings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp- 1764-1773. PMLR, 2019.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International conference machine learning, pp. 2943-2952. PMLR, 2020.
Farzan Farnia, Jesse M. Zhang, and David Tse. Generalizable adversarial training via spectral nor- malization, 2018
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neu- ral networks. In 7th International Conference on Learning Representations, ICIR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks, 2019.
Scott Gray, Alec Radford, and Diederik Kingma. Gpu kernels for block-sparse weights. 2017. URL htttp:://cd..oree
Suriya Gunasekar, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Sre- bro. Implicit regularization in matrix factorization, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 14:12876-12889, 2021.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.0633: 2019.
Erdong Hu, Yuxin Tang, Anastasios Kyrillidis, and Chris Jermaine. Federated learning over images: Vertical decompositions and pre-trained backbones are difficult to beat, 2023. URL https:
//axxiv.org/ab///222
03237
Tiansheng Huang, Shiwei Liu, Shen, Fengxiang He, Weiwei Lin, and Dacheng Tao. Achieving personalized federated learning with sparse local models. CoRR, abs/2201.113800 2022.
Dzmitry Huba, John Nguyen, Kshitiz Malik, Ruiyu Zhu, Mike Rabbat, Ashkan Yousefpour, Carole- Jean Wu, Hongyuan Zhan, Pavel Ustinov, Harish Srinivas, Kaikai Wang, Anthony Shoumikhin, Jesik Min, and Mani Malek. PAPAYA: practical, private, and scalable federated learning. In Diana Marculeseu, Yuejie Chi, and Carole-Jean Wu (eds.), Proceedings of the Fifth Conference Machine Learning and Systems, MLSys 2022, Santa Clara, CA, USA, August 29 September 1, 2022. mlsys.org. 2022.
Siddhant M. Jayakumar, Razvan Pascanu, Jack W. Rac, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training, 2021.
Geonhwa Jeong, Po-An Tsai, Abhimanyu Rajeshkumar Bambhaniya, Stephen W. Keckler, and Tushar Krishna. Abstracting sparse DNN acceleration via structured sparse tensor decomposi- tion. CoRR, abs/2403.07953 2024.
12

---

Published as a conference paper at ICLR 2025
Yuang Jiang, Shiqiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K. Leung, and Leandros Tassiulas. Model pruning enables efficient federated learning on edge devices. IEEE Trans. Neural Networks Learn. Syst., (1((1))11374-103388 2023. doi: 10.1 1O0.OOONNS
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista A. Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L.. D'Oliveira, Hubert Fichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Gar- rett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons. Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He. Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Hang Qi. Daniel Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang. Li Xiong. Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning, Found. Trends Mach. Learn., 14(1-2):122) 2021. doi: 10. 100.161/2200000083.
Mikhail Khodak, Renbo Tu, Tian Li, Liam Li, Maria-Florin Balcan, Virginia Smith, and Ameet Talwalkar. Federated hyperparameter tuning: Challenges, baselines, and connections to weight- sharing. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jen- nifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 19184-19197, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade. and Ali Farhadi. Soft threshold weight reparameterizztion for learnable sparsity. In International Conference on Machine Learning, pp. 5544-5555. PMLR, 2020.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Inderjit S. Dhillon, Dimitris S. Papail- iopoulos, and Vivienne Sze (eds.), Proceedings of the Third Conference Machine Learning and Systems, MLSys 2020, Austin, TX. USA, March 2-4, 2020. misys.org, 2020a.
Xiang Li, Kaixuan Huang, Wenhao Yang. Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRevvew.nee, 2020b.
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations, 2019.
Zi Lin, Jeremiah Zhe Liu, Zi Yang, Nan Hua, and Dan Roth. Pruning redundant mappings in transformer models via ppectral-normallee identity prior, 2020.
Junjie Liu, Zhe Xu, Runbin Shi. Ray C. C. Cheung, and Hayden Kwok Hay So. Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers. In 8th Interna- tional Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Christos Louizos, Karen Ulirich, and Max Welling. Bayesian compression for deep learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnet (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp.
2017.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through 1.0 regularization. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 -May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
13

---

Published as a conference paper at ICLR 2025
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in rein- forcement learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Ávila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning, ICML 2023, 23-29 July 2023, Honolull, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 23190-23211. PMLR, 2023.
Amirhossein Malekijoo, Mohammad Javad Fadacieslam, Hanieh Malekijou, Morteza Homayoun- far, Farshid Alizadeh-Shabdiz, and Reza Rawassizadeh. Fedzip: A compression framework for communication--ffcceent federated learning. arXiv preprint arXiv: arXiv:2102.01552 10230/99,, 2021.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In CVPR, pp. 7765-7773, Computer Vision Foundation/ / IEEE Computer Society, 2018.
Brendan McMahan et al. Communicationn-fficccnn learning deep networks from decentralized data. In Artificial intelligence and statisties, pp. 1273 1282.. PMLR, 2017.
Yiqun Mei, Pengfei Guo, Mo Zhou, and Vishal Patel. Resource-adaptive federated learning with all-in-one neural composition. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho. and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Con- ference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 9(1):2383, 2018
Dmitry Molchanov, Arsenii Ashukha, and Dmitry P. Vetrov. Variational dropout sparsifies deep neural networks. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th Interna- tional Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp.
PMLR, 2017.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pp. 4646-4655. PMLR, 2019.
Yue Niu, Saurav Prakash, Souvik Kundu, Sunwoo Lee, and Salman Avestimehr, Federated learning of large models at the edge via principal sub-model training. CoRR, htss/200..11141 2022. doi: N0.48550/ARXXI.2208. 13141.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein. Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Z. Yang. Zachary Veti,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Informa- tion Processing Systems 32: Annual Conference on Neural Information P Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8024-8035, 2019.
Sarthak Pati, Ujjwal Baid, Brandon Edwards, Micah Sheller, Shih-Han Wang. G Anthony Reina, Patrick Foley, Alexey Gruzdev, Deepthi Karkada, Christos Davatzikos, et al. Federated learning enables big data for rare cancer boundary detection. Nature communications, 13(1):7346, 2022.
14

---

Published as a conference paper at ICLR 2025
Xinchi Qiu, Javier Fernandez-Marge Pedro PB Gusmao, Yan Gao, Titouan Parcollet, and Nicholas Donald Lane. Zerofl: Efficient on-device training for federated learning with local sparsity, 2022. htthttp:: http::://aaxvvooooo abs/ /a/0//222...2 02507
Md Aamir Raihan and Tor M. Aamodt. Sparse weight activation training, 2020. URL https:
/ //arxiv.org/aaa abs 22012001110000000
Swaroop Ramaswamy, Rajiv Mathews, Kanishka Rao, and Françoise Beaufays. Federated learning for emoji prediction in mobile keyboard. CoRR, 0es//100604329) 2019.
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In 9th Inter- national Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Tim Salimans and Durk Kingma. Weight normalization: simple reparameterization accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
Felix Sattler, Simon Wiedemann, Klaus-Robert Müllrr, and Wojeiech Samek. Robust and communication-efficcent federated learning from non-iid data. IEEE transactions on neural net- works and learning systems, 2019.
Jonathan Schwarz, Siddhant M. Jayakumar, Razvan Pascanu, Peter E. Latham, and Yee Whye Teh. Powerpropagaation: sparsity inducing weight reparameterisation, 2021. URL https:
www.v..og.abb///....o.. 200200009
Esha Singh, Shane Bergsma, Nolan Simran Dey, Joel Hestness, and Gavia Gray. Empirical upper bounds for unstructured sparsity in compute-efficien language modeling. In Workshop Ma- chine Learning and Compression, NeurlPS 2024, 2024. URL httpps://ppenrevieew.. net/
Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel R. D. Rodrigues. Generalization error of invariant classifiers, 2017.
Tomas Vaskevicius, Varun Kanade, and Patrick Rebeschini. Implicit regularization for optimal sparse recovery. CoRR, abs/1909.05122. 122, 2019.
Ewen Wang, Boyi Chen, Mosharaf Chowdhury, Ajay Kannan, and Franco Liang. FLINT: A plat- form for federated learning integration. In Dawn Song, Michael Carbin, and Tianqi Chen (eds.), Proceedings of the Sixth Conference Machine Learning and Systems, MLSys 2023, Miami, FL, USA, June 4-8, 2023. mlsys.org, 2023.
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H. Brendan McMahan, Blaise Agüera y Ar- cas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, Suhas N. Diggavi, Hubert Eichner, Advait Gadhikar Zachary Garrett, Antonious M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He, Samuel Horváth, Zhouyuan Huo, Alex Ingerman, Martin Jaggi, Tara Javidi, Peter Kairouz, Satyen Kale, Sai Praneeth Karimireddy, Jakub Konečný, Sanmi Koyejo, Tian Li, Luyang Liu, Mehryar Mohri, Hang Qi, Sashank J. Reddi, Peter Richtárík, Karan Singhal, Virginia Smith, Mahdi Soltanolkotabi, Weikang Song, Ananda Theertha Suresh, Sebas- tian U. Stich, Ameet Talwalkar, Hongyi Wang, Blake E. Woodworth, Shanshan Wu, Felix X. Yu, Honglin Yuan, Manzil Zaheer, Mi Zhang, Tong Zhang, Chunxiang Zheng, Chen Zhu, and Wennan Zhu. A field guide federated optimization. CoRR, abs/2107.0691 2021.
Pete Warden. Speech commands: dataset for limited-vocabulann speech recognition. arXiv preprint arXiv:1804.032000 2018
Peter Welinder, Steve Branson, Takayuki Mita, Catherine Wah, Florian Schroff. Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology. 2010.
CUB 200.000000 mtml
Xueyu Wu, Xin Yao, and Cho-Li Wang. Fedser: Structure-basse communication reduction for federated learning. IEEE Transactions on Parallel and Distributed Systems, 32(7):155--1577, 2020.
15

---

Published as a conference paper at ICLR 2025
Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ra- mage, and Françoise Beaufays. Applied federated learning: Improving google keyboard query suggestions. CoRR, abs/1882.002033, 2018.
Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander Rush, and Yann LeCun. Adversarially regularized autoencoders. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 5902-5911. P0L...991............ 10-15 Jul 2018.
Peng Zhao, Yun Yang, and Qiao-Chu He. High-dimensional linear regression via implicit regular- ization. Biometrika, 10344400,33 February 2022. ISSN 1464-3510. doi: 10. 10.109//ioomet/ asac010.
16

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION
We implemented all experiments using a ResNet-18 backbone (He ct al. 2016), which was adjusted to accommodate the number of classes for each task. The training pipeline developed using PyTorch (Paszke et al] and specific components of the various methods, such as layer-wise activation pruning and weight re-parameterization, were implemented through custom layers and functions.
To simulate the federated learning environment, we used the Fl Flower framework (Beutel et al., 2022). All models were trained from scratch, without pre-trained weights, to ensure a fair evaluation of SparsyFed's performance under different sparsity settings. We only performed fine-tuning of a pre-trained model for the experiments on Visual Transformer (ViT) models.
We utilized a template from CaMLSys Lab to structure the code and ensure easy reproducibility. The complete code for the implementation is available in this repository
B RELATED WORK
Sparse training in centralized settings. Methods to enforce sparsity in neural networks can be grouped into two main categories: (1) dense-to-sparse methods that train dense model and achieves
a sparse model after training (Molchanov et al. 2017 Louizos et al. (017), (2) sparse-to-sparse
methods where pruning happens during training (Louizos et al. 2018 Dettmers Zettlemoyer, 2019 [Evi et al. 2020) Jayakumaa et al. [2021; Raihan & Aamodt 2020). thus theoretically reducing
computational requirements (Bengio et al.] 2015) and speeding up training. Our work introduces a novel method inspired by sparse-to-spane approaches in centralized settings such as Raihan &
Aamodi (2020) to implement sparse training directly within FL clients.
Pruning model updates in FL.. After-training model pruning, where clients regularly train their dense model and then apply pruning to the resulting updates, has been widely explored in the FL lit- erature (Sattee et al. 2019: Wu et al. 2020; Malekijoo et al.) 2021). /Stttle et al. (2019) introduced Sparse Ternary Compression, which combines Top-K pruning with ternary quantization on client updates. Similarly, FedZip (Malekijoo al. 2021) applies layer-wiss pruning approach, while FedSCR (Wu et al.] (2020) leverages patterns in client updates for more aggressive compression. However, these methods primarily focus on enhancing communication efficiency without tackling the issue of reducing computational overhead during training, as they conduct regular training on a dense model before pruning weight updates. Our approach simultancously addresses both concerns, as training sparse model naturally yields sparse updates. Furthermore, existing after-training prun- ing methods typically focus solely on upstream communications or experience performance degra- dation when applied downstream. In contrast, our method effectively maintains sparsity after server- side aggregation, as clients rapidly converge on a shared sparsity mask, ensuring that server-to-client payloads remain sparse.
Sparse training in FL. Few studies have investigated the benefits of sparse learning in federated
settings (Bibikar et al. 2022 Huang et al.] 2022 Jiang et al. 2023 Qiu et al. 2022 Babakniya al. 2023). Specifically. FedDST (Bibikar et al.
2022) used RigL. (Evei et al. (2000) to train sparse
models on clients but mainly focused on highly heterogeneous data distributions without considering extreme sparsity levels. FedSpa (Huang et al. 2022) employed a randomly initialized sparse mask that remained fixed throughout training, providing no clear justification for this approach. PruneFFL (Jang et al. 2023) computes the sparse mask during the initial round on a designated client using its potentially biased data. Qiu al. (2022) empirically found that, after local training, the Top-K weights differ across clients, particularly at higher sparsity levels, leading to divergent sparse masks. This divergence makes aggregation inefficient and results downstream dense models. In response, Qiu et al. (0222) proposed ZeroFL, which integrates unstructured SWAT during local client training- However, this alone does not guarantee achieving the desired sparsity level, as SWAT often leads to weight regrowth with each optimizer step. To address this, ZeroFL applies Top-K pruning before sending the model back to the server, ensuring the model meets the targeted sparsity. is essential to highlight that ZeroFL applies the same sparsity level uniformly across all model layers, regardless of their sensitivity. The recent work in Babakniya et al. (2023) introduces FLASH, which employs fixed binary mask throughout the training process. This mask is established during a warm-up
17

---

Published as a conference paper at ICLR 2025
phase where subset of clients ([10, 20)) train their model for non-negligible number of epochs ([10, 20, 40]), and compute their per-layer sensitivity. The server aggregates the client's 's sensitivities to determine the mask. This mask's s fixed nature means no weight regrowth is allowed, and no further pruning is required after the warm-up phase.
Cross-device federated learning. Federated Learning (FL) has emerged as a paradigm shift from traditional machine learning approaches, improving data privacy and moving the computation load to the network's edge. FL enables collaborative model training across decentralized devices while keeping data localized, thereby mitigating risks associated with centralized data aggregation (McMahan et al. 2017). In FI., client devices participate in model training via iterative rounds, aggregating local updates to build a global model. This distributed approach is advantageous for scenarios involving edge devices with limited computational resources and intermittent connectiv- ity (Kairouz al., 202T). Cross-device Federated Learning (FL) particularly challenging due to nature of client devices, such smartphones and IoT devices. The variability in hardware capabilities and data distributions across devices necessitates specialized techniques to optimize computation and communication techniques, such as quantiza- tion and model pruning, which have shown promise reducing the amount of data that must be transmitted during each communication round. However, these methods often face challenges in maintaining model performance and achieving consensus on the sparsity patterns among clients.
the heterogeneous and essource-constralie
Model pruning. Sattler et al. (2019) propose Sparse Ternary Compression (STC), lossy com- pression scheme able to reduce the per-round communication cost of FL iterations significantly. STC first applies Top-X pruning after unaltered on-device training and then further compresses the weight updates using ternary quantization. Other types of work have followed the idea proposed in Rigging the Lottery Ticket (Evci et al., 2020), where an initial sparse mask fixed at the beginning of the training and remains primarily unchanged throughout. This approach allows clients to train only the initially fixed weights (Jiang et al. 2023) Babakniya et al. (202)).
Weight parametrization. Re-parameterization of weights in machine learning refers to the pro- cess of altering the re-parameterization of a model's weights to achieve various goals or facilitate specific training strategies (Li et al.) 2019; Gunasekar et al.) 2017 Zhao et al. 2022, Vaskevicius et al. 2019). This approach can be beneficial for enhancing the model's robustness to the applica- tion of sparsity. This work considers a re- parameterization based on Weight Spectral Normalization (Miyato et al. 2018) and Powerpropagation (Schwarz et al.) 2021). The former re- parameteriize the weights based on the proportion of each weight relative to the one with the highest magnitude. At the same time, the latter applies an alpha power the weights, inducing "rich get richer" mechanism.
Wrefactor = W (W)
C ADDITIONAL EXPLANATIONS ON SparsyFed COMPONENTS
Model preparation for pruning. The first step is to prepare the model for pruning by applying re-parameterization to the weights at the layer level, proposed in /Schwarz et al. (202T). This approach leverages the information already present in the model by applying power to a specific value, 8, to the network weights. This re- parameterrization aims to induce sparse representation of the weight of the network.
Following the original approach, each weight wy is replaced by w." This transformation only applies the neural network weights, leaving other parameters unchanged. Given the re-parameterized loss function L( L(w²), the gradient with respect to w becomes:
V L(w") Vwe L(w") B-1
Here, V L(w") is the gradient concerning the re- parameterized weights. This gradient is scaled element-wee wise by (w.3-1 which adjusts the step size proportionally to the magnitude of each weight. This update is distinct from simply scaling the gradients in the original re-parameterization, as it
18

---

Published as a conference paper at ICLR 2025
directly modifies the re-parameterized weights, not the original weights. This modified gradient step enables a "rich-get-richer" dynamic where the gap between high and low low-manittude weights increases, creating a natural separation between them. This makes the model relatively insensitive to pruning.
Weight pruning. Our first concern was to reduce the model's size during communication rounds speed up FL training. In this context, compression mechanism is typically used reduce the payload size that has to be exchanged. To achieve this, reducing the number of parameters in the network is necessary, inducing high level of sparsity. We used a Top-K pruning method to remove all low-magnitude weights from the network. This pruning operation is implemented at the end of each local training on the client. This decision is based on several observations: (a) since the parametrization affects gradient descent, pruning before training would not be beneficial, (b) the server is not aware of the features of the data on the clients, and initializing the sparsity mask on the server side could cause significant drop in performance.
We also decided to induce sparsity from the very first round. The clients would only use a small portion of the weights during training, they must start training with the restricted fraction to maximize their effectiveness. Sparsity is induced from the first round of local training on each device, allowing the local updates to be sparse from the beginning of the federated training.
During the following training rounds, a second quality of Powerpropagation comes into play, dras- tically reducing the regrowth of new weights during training. This results in minimal shifts of the model's sparsity mask from the global one received from the server. Thus, the sparsity mask of the new model update sent to the server will differ very little from the global one. In other words, using Powerpropagation, we are forcing all clients to converge towards a shared sparsity mask, similar Evei et al. (2020), without inhibiting the regrowth of new paths. The clients decide the mask in the first round of training based on their local data. This means that the global model will remain highly sparse even after aggregating new local updates from the clients. This result allows for signif- icant compression during the downlink communication from the server to the clients in subsequent training rounds.
Activation pruning. To further reduce the computation footprint during the training, we imple- mented layer-wise pruning similar to the one proposed in Raihan & Aamodi (2020). The original proposal was to speed up inference and training by reducing the number of computational opera- tions, inducing fixed level of sparsity the weights of each layer during both the forward and backward passes. However, this levels, as not all layers retain the same level of information. Applying the same sparsity all layers could negatively impact those that naturally retain more information. In our implementation, we heavily modified the approach, retaining only key concepts. Since we keep the model sparse during almost all training rounds, except for the first, pruning the weights during the forward pass is not necessary.
cause a significant drop in performance with high sparsity
For the pruning of activations during the backward pass, instead of applying the same level of spar- sity to all layers the model, the global sparsity mask used to determine the pruning sensitivity of each layer. The level of pruning applied to the activations is directly proportional to the level of the sparsity the weights in the same layer, allowing layers that retain more information to main- tain denser activations while drastically reducing the activations in layers that reach high level of sparsity.
D ADDITIONAL BACKGROUND
D.1 POWERPROPAGAATOON
Powerpropagation is weight re-parameterization technique that induces sparsity in neural net- works. Essentially, it causes gradient descent to update the weights in proportion to their magnitude, leading to "rich get richer" dynamic where small-valued parameters remain largely unaffected by learning. As a result, models trained with Powerpropagation exhibit a weight distribution with significantly higher density at zero, allowing more parameters to be pruned without compromising
19

---

Published as a conference paper at ICLR 2025
performance. Powerpropagatton involves raising model parameters to the power of 3 (where 1) in the forward pass while preserving their sign. This re-parameterization can be expressed as
|0|8-1
w
where w represents the weights, and v are the re- pe-paammeterieed parameters. Due to the chain rule, this re-parameterizatioon causes the magnitude of the parameters (raised to 8 1) appear in the gra- dient computation. Consequently, small-valued parameters receive smaller gradient updates, while vageev parameters receive larger updates, thus amplifying the "rich get richer" dynamic. Pow- crpropagation leads to intrinsically sparse networks, meaning that a significant portion of the weights converge values near zero during training. This property allows for the pruning (removal) of many weights without significantly compromising model performance. Powerpropagation can also be easily integrated with existing sparsity algorithms enhance performance further. Studies show the benefits of combining Powerpropagation with popular methods such as Iterative Pruning and TopKAST (Gale et al. 2019: Jayakumar al. 2021
D.2 FIXED MASK SPARSE TRAINING (FLASH)
Training fixed subset of weights can be practical approach for sparse model training in Feder- ated Learning (FL), offering several significant benefits. Limiting the number of active (non-zero) weights reduces the computational and memory demands compared to dense models, which is par- ticularly advantageous given the resource constraints often found in client devices within FL. In ad- dition to these resource savings, sparse models enhance communication efficiency between clients and servers. Since only the active weights need to be transmitted, the message size is reduced, leading to faster training and lower bandwidth usage.
Another advantage this method its potential to uncover "winning tickets" within neural net- works Frankll works[Frankle & Carbin (2019). Research indicates that dense, randomly initialized networks often contain sparse sub-networks, known as "winning tickets", which, when trained independently, can achieve performance similar to the full model. Training a fixed subset of weights promotes this sparsity and encourages the model to learn more efficient representations, potentially revealing these sub-networks.
Despite these benefits, specific challenges arise when using fixed subset of weights in FL. If the mask is not initialized correctly or fails to adapt during training, the model's performance may be compromised. To mitigate these issues, strategies such sensitivity-baser pruning and selective mask updates are crucial for fully leveraging the advantages of sparse learning in FL Babakniya et al. (023). The training process for sparse models can be formalized as:
W sparse M O W
where W Ssparse represents the sparse weights, M is the binary mask (with values 0 or 1), and @ denotes element-wwi wise multiplication.
D.3 SPECTRAL NORMALIZATION
Spectral Normalization (SN) is weight normalization technique that aims to limit the most signif- icant singular value of each weight matrix, thereby controlling the Lipschitz constant of the func- tion represented by the network. This is achieved by constraining the spectral norm of each layer : hin - hout- Formally, given a weight matrix W, its spectral norm is defined as the most significant singular value (W), i.e.,
|hil2
(W)- = max
h
|/1|
SN normalizes the weight matrix by dividing by its spectral norm:
W
W
(WW)
20

---

Published as a conference paper at ICLR 2025
This ensures that the Lipschitz constant of each layer remains bounded, promoting stability during training, particularly in scenarios where gradients may explode. Originally proposed as regular- ization technique to stabilize discriminator training in Generative Adversarial Networks (GANs) Miyato et al. 2018), SN has since found broader applications, including improvements genera- tive neural networks Zhao al. (2018), and has been theoretically linked to enhanced generalization and adversarial robustness Farnia et al. (2018); Sokolic et al. (2017); Cisse et al. (2017).
In the context of federated learning and sparse training, SN can significantly improve the resilience of models to pruning. SN regularizes the model mappings by enforcing the Lipschitz constraint, reducing sensitivity high sparsity levels. This effect has been observed in prior work on pruning, where SN was used to prune redundant mappings and enforce a spectral-normalized -nrmmalized identity prior Lin et 2020).
However, by strictly constraining the Lipschitz constant, SN may reduce the model's flexibility during training, potentially affecting convergence some cases. To address this, following an approach similar to those proposed in other re-parameterization works Vaskevicius et al. (2019), propose a slightly different method where the spectral norm is modified to enhance the distribution of the weights. Specifically, we redefine the weight update rule as:
This approach allows us maintain most of the model's 's performance while improving its resilience to pruning.
21

---

Published as a conference paper at ICLR 2025
E ADDITIONAL EXPERIMENTAL RESULTS
E.I NAIVE POWERPROPAGATION FEDERATED ADAPTATION
In the original Powerpropagation paper, pruning was applied at the end of centralized training after training a dense model. To create baseline for comparison, we implemented naive federated version of Powerpropagation that follows this methodology. In this version, pruning is not applied during local training at the client level. Instead, clients train with the full model, sending and receiv- ing full model updates. At the end of the federated training (after the final round), pruning is applied to the global model using the Top-K method.
We compare this naive version of federated Powerpropagation to our proposed method, SparsyFed, across various sparsity levels and different levels of data heterogeneity, in Table 3 As shown in the results, SparsyFed significantly outperforms the naive federated Powerpropagation in terms of performance and stability, even in sparsity.
10a (IID)
1.0 (non-IIID)
0.1 (non-IIID)
Sparsity
Naive PP
SpursyFed
Naive
SparsyFed
Naive
SparsyFed
0.000
84.69 t 1.57
84.31 + 1.01
74.86 2.28
0.900
71.41 + 7.01
84.31 + 0.86
66.70 + 3.76
82.13 + 1.53
45.82 6.32
75.00 2.78
0.950
35.02 + 6.37
84.25
1.38
33.84
13.71
82.60 + 1.58
24.28
8.79
75.95
3.39
0,990
12.40::.0
77.16
0.85
12.30 : 3.68
77.71 + 70.01 + 0.43
+ 2.52
63.69 3,90
0.995
10.33 + 0.57
72.71
10.03 + 0.05
9.74 + 3.24
56.79
3.97
0.999
9.86 +: 0.25
55.24 + 2.09
10.01 4 0.02
51.39 + 3.19
11.67
6.66
43.68 + 7.61
Table 3: Accuracy comparison for Naive Powerpropagation and SparsyFed on CIFAR-10 with dif- ferent LDA settings (a = 10 = 1.0, and a = 0.1).
E.2 POWERPROPAGATION EXPONENT IN SparsyFed
Applying Powerpropagation in SparsyFed introduces a new hyperparameter that must be tuned alongside others. To address this, we explored different approaches. The first follows the methodol- ogy of the original paper, where series of fixed exponents were proposed for re-parameterization In the latter approach, ing it dependent on the network's weights rather than being predefined. This follows a strategy similar to spectral normalization.
propose a novel method for determining the exponent dynamically, mak-
E.2.1 SENSITIVITY ANALYSIS OF FIXED POWERPROPAGAATOO EXPONENT
Following the approach of the original paper, we evaluated different values of 3 to assess their impact on model performance. Preliminary tests suggest that the sensitivity of this hyperparameter is not as critical as initially expected. We tested several values proposed in the original paper to evaluate their effectiveness in sparse training within a federated learning setting. As shown in Fig. 5 any B value between and 2 significantly improves performance in dynamic sparse training compared to the baseline without re- parameterization.
E.2.2 HYPERPARAMETER--FE POWERPROPAGATION EXPONENT
To eliminate the need for an additional hyperparameter, we propose an alternative version of SparsyFed where 3 is computed at runtime. Instead of a fixed 33 for Powerpropagation, tailored ex- ponent is derived based on the layer-wise weight magnitude distribution, leveraging concepts from the spectral norm.
We denote the spectral norm a weight matrix W as spectral W), which computes the maximum magnitude of the elements in W and normalizes it by dividing W by its maximum value W mmax- Thus, spem () results in a matrix of positive values ranging between 0 and 1. Using pnoectralnormm/ WW, construct tailored exponent matrix for the network's 's weights by implementing custom convolutional and linear layers, where the matrix is computed at the beginning of the forward pass. The exponentiation process raises each weight in layer the power of
22

---

Published as a conference paper at ICLR 2025
1 + ppectral.normi/e ()), ensuring that each weight is scaled by factor between 1 and proportional to its relative magnitude compared to the largest weight in the network.
To reduce computational overhead, the exponent matrix is calculated only during the first forward pass and stored for subsequent iterations. Additionally, to minimize memory usage, we avoid storing the full exponent matrix and instead compute and store only the average value per layer, reducing memory requirements to a single scalar per layer. This averaging approach results in higher values for denser layers with more nonzero weights, while highly sparse layers tend have significantly smaller values. The sparsest layers, often found toward the end of the network, tend to be larger and more sparsely populated, amplifying this effect.
The overall computational cost remains marginal and comparable to the standard Powerpropaga- tion implementation, as the computation is only performed during the first forward pass. Perfor- mance analysis in Fig. X shows that this method outperforms naive Top-K pruning without re- parameterization and matches or surpasses most fixed alpha values. However, it still falls short of the best- cesttperformin fixed 3, suggesting that further refinements would be necessary to bridge this performance gap. While more extensive experiments are needed, this represents a useful insight into potential enhancements for this approach.
90
85
Test Accuracy (%)
80
75
70-
Resee188 (Desse)
Spectral Exponeat
SpangFed
1.25
2.50
2.75
Exponent
Figure 5: Test Accuracy with different 33 values, with 95% sparsity CIFAR-10 LDA The accuracy of the dense model (gray), the hyperparameter-free Spectral Exponent version, and the Top-K K method are also reported for reference.
1.0.
E.3 WEIGHT MOVEMENT METRICS IN FEDERATED LEARNING
To better understand the aftermath of SparsyFed on the weight of the model during training, a com- prehensive evaluation of the weight evolution throughout training has been conducted by comparing the initial global model, the local updates sent to the server at the end of each round, and the sub- sequently obtained aggregated global model. This allows for gated metries that capture the overall training progression and the round-specing dynamics. We focus on two primary metrics: the L2 norm between weight matrices and cosine similarity. The former (L2) quantifies the magnitude of weight changes. It is measured as follows: (i) the cumulative L2 norm between the initial model and the aggregated model at each round for long-term evolution, and (ii) the L.2 norm between consecu- tive global models to assess round-level variation. In contrast, cosine similarity captures functional consistency in the updates. alignment across non-lID data distributions and (ii) between consecutive global models to evaluate directional stability in weight updates.
is measured as (i) the similarity between client updates to assess
Experiments were conducted using ResNet18 CIFAR-10, CIFAR-100 and Speech Commands, with data partitioned using LDA (a = 0.1) to simulate non-IID distributions. The model was trained for 200 rounds using SparsyFed and Top-K under sparsity levels of 95%. The results reveal distinct training behaviors across methods.
As shown in Fig. 6 the global L2 distance measuring the deviation from the initial increases in SparsyFed. whereas in other implementations, rises sharply before stabilizing after several rounds, indicating excessive drift in the early stages. In contrast, SparsyFed
23

---

Published as a conference paper at ICLR 2025
CIFAB.10 10
CIPAR.10
Sperd Cormerds
SperRa
123 Disaaa
3-6
20
140
200
100
Ileends
Strezda
Figure 6: Global L2 Norm- This metric tracks how far the global model moves from its initial state throughout training. A smoother increase indicates more stable updates, while sharp rises suggest rapid drift.
12
Ilaends
Jirezdd
Figure 7: Round L.2 Norm - By measuring the 1.2 norm between consecutive global models, this metric highlights the magnitude of updates applied at each training round, revealing how steadily or abruptly the model evolves.
exhibits gradual trajectory, suggesting stable and incremental updates. A similar trend is observed when examining round-by-round evolution, highlighting the steps taken by the model af- ter each aggregation. Fig. 7 illlsstrates the high variance in updates for Top- K. whereas SparsyFed maintains stable and smaller updates.
CIPAS-40
CIEAR.00
1.00
0.1
Chetre Stetlurky
500
Bleaads
Roradda
Figure& 8: Round Cosine Similarity - This captures the consistency of global model updates across rounds. A high and stable similarity suggests smooth training dynamics, while fluctuations may indicate instability in model aggregation.
Cosine similarity metries provide complementary insights. The round-wise cosine similarity (Fig. () represents the cosine similarity between the global model before local training and after aggregation. This measurement shows how much the model is modified by training round. The results highlight that SparsyFed quickly attains high alignment and maintains stability throughout training, while Top-K exhibits more significant fluctuations in the initial rounds, suggesting inconsistent updates across rounds.
Figure g shows client round-wise cosine similarity, which measures the similarity among updates sent to the server by clients at the end of each training round. The results indicate that SparsyFed tends to have faster and smoother trajectory. both cases-client and round- all methods eventu- ally achieve high similarity, though at different stages of training. Notably, SparsyFed demonstrates the fastest convergence and smoother dynamics. A slightly lower final similarity value in the CIFAR- 10 experiment is not concerning, provided it remains consistent and stable throughout training, as this may indicate better handling of data heterogeneity, allowing clients to adapt more effectively to their respective data distributions.
24

---

Published as a conference paper at ICLR 2025
CIEAR.00
Speec Germards
000
2.55
Chatte
023
6.3
009
30
630
150
Bleeadss
Hinzada
Figure 9: Client Cosine Similarity - This measure assesses how similar client updates are. Higher similarity suggests greater alignment across different clients, while lower values indicate di- verse local updates due to data heterogeneity.
E.4 GLOBAL MODELS SPARSITY LEVELS
The global model's sparsity fluctuates during training due to the mismatch in client updates. As a result, the global model's sparsity is not always fixed and fluctuate significantly throughout the training process. The following figures show that the sparsity target directly influences the sparsity level. Higher sparsity targets tend lead more significant weight regrowth during training, as seen in Fig. 12, which results in more considerable mismatch on the server, leading to a denser model. This effect is particularly noticeable the beginning of training, when the model is more susceptible to significant changes in shape, as illustrated in Fig. 3 Following this, we present a plot of the sparsity measure on the global model for different sparsity targets during training.
96-
Sparsty (96)
90-
Spanity (50
96
85
94-
SpsneyFol
SpansFee
Tup-
Tip-K
Trgget
Targe Spensite
90
200
3U11
500
700
2001
400
500
6000
700
100
Spusitt (16)
Spansity (95)
ZenFLL
Spans Peci Top-K
Speny Fed Top-R
(00
3011
5000
700
Romnd
(a) From the top left the bottom right, we present the plots of the sparsity levels for ZeroFL, SparsyFed, and Top-K at sparsity levels of 95%, 99%, 99.5%, and 99.9%. ZeroFL struggles reach the target sparsity in all cases, partially due its aggregation method, which only aggregates non-zero weights. This leads to large magnitude weights, even if they are present in only a fraction of the clients. SparsyFed and Top-K tend to reach the target more quickly, with SparsyFed showing a small fluctuation in the initial training phase due to the movement of the mask, as shown in Fig. 3
Figure 10:
E.5 DISTRIBUTION THE SPARSITY THROUGH THE LAYERS.
Sparsity distribution is crucial as the sparsity achieved in the weights of each layer is used to de- termine the sparsity level applied to the activations during the backward pass. Each layer pruned with distinct sparsity level based the information contains. leveraging layer sensitivity to implement an effective pruning strategy for activations. This ensures that the sensitivity observed in the weights is reflected in the activation pruning, allowing for dynamic sparsity for both weights
25

---

Published as a conference paper at ICLR 2025
1.0
0.8
Sensitivity
0.6
0.4
convl
layer3
fc
layer1
layerd
ZeroFL
0.2
0.0
20
40 Layer Index
80
100
Figure 11: Different sparsity ration of some relevant layer, ZeroFL. (red) vs SparsyFed with Pow- erpropagation and sparsity ratio of 60%. The first layers ZeroFL. Empirical observation regarding the nature of the pruning procedure (constant across lay- for ZeroFL, variable unstructured across layers for SparsyFed), We could use this in the main paper only if we make the case that our method performs better because of this. We consider this appendix material for now.
shown they kept dense in
and activations based on the natural sensitivity each layer. Empirical evidence, shown in Fig. II supports this approach, showing that in ZeroFL implementations, the sparsity of the layers remains uniform across all layers of the model, while in others, the sparsity levels vary significantly between layers. The first layers tend to be nearly fully dense, while the deeper layers exceed the global sparsity target, indicating that the first layers retain more information than the deeper layers.
A key factor in this behavior is the limited weight regrowth observed in SparsyFed, where only small number of weights transition from zero to non-zero values after each training round. As de- picted in Fig. [2] SparsyFed exhibits minimal regrowth, stabilizing quickly over a few rounds. This is directly attributable to using Powerpropagation during training, drastically reducing the impact of smaller weights. Although this behavior was not highlighted the original paper, represents empirical evidence supporting the effectiveness of an inherited sparse model training procedure.
As illustrated Fig, 3 our implementation shows that the global model's 's mask shifts slightly during the initial training rounds. This suggests that overly rigid approaches, such as those proposed in FLASH, may negatively impact performance by failing to accommodate necessary flexibility. On the other hand, SparsyFed's approach maintains flexibility while still converging toward a consistent global mask.
E.6 HETEROGENEOUS SPARSITY EXPERIMENTS
In this experimental setup, we aim evaluate heterogeneous sparsity by using two distinct sets of model sparsity:
0.99, 0.85, 0.8]: These sparsity levels are based on the settings proposed by FLASH in their heterogeneous setup. These values represent moderate range of sparsity, which does not significantly impact the model's ''s performance in this task.
2. (0.99, 0.95. 0.9] These denser models previously tested in other experiments offer more challenging setup. SparsyFed outperforms FLASH in these settings due to its ability to adapt efficiently varying sparsity levels across clients.
26

---

Published as a conference paper at ICLR 2025
10
Weight Regrowth
10
ZeroPL
Spansy
Model Size
Target Density
10a
100
200
300
400
500
600
700
Rounds
Figure 12: Number of weights regrown (weights switching from zero to non-zero) with one local epoch, compared between SparsyFed and ZeroFL.. ZeroFL performs the worst of all tested imple- mentations, with 50-80% of weights regrowing after local training. In contrast, SparsyFed shows near-zero regrowth (200x lower than ZeroFL). This ensures that the movement of the mask fo- cuses on important weights rather than being distributed across many. Note: FLASH does not allow weight regrowth due to ixek-mask training approach, excluded from this graph.
The clients partitioned into three groups containing (40, 30, 30] clients. Each group trains different level of sparsity in the model. They receive a model matching their capability and send back update of the dimensions. For clarity, client in group 0 in the first setting will receive a model with 0.9, train on it, and then send back a sparse update with 0.9. As shown in Table 4 we achieve high performance in all the settings, with clear margin in the more extreme setting
Lower Sparsity
Higher Sparsity
Setting
LDA
FLASH
SparsyFed
FLASH
SparsyFed
a 103
83.15 + 0.96
83.28 + 0.44
83.77 0.84
83.27 0.44
Moderate
a 0.1
70.9 + 0.92
74.97 + 2.39
74.65 = 0.96
74.98 + 2.39
100
74.72 + 0.49
81.04 + 0.26
74.72 = 0.67
81.20 + 0.39
Extreme
a = 0.1
57.63 4.83
69.74 + 0.28
59.82 = 2.99
69.96 0.19
Table 4: Performance comparison between FLASH and SparsyFed across different heterogeneous sparsity settings and LDA values. The model has been evaluated on all the sparsity-level trained. For simplicity, we show the accuracy for the denser model (which reaches a density equal to the target density trained, 0.8 and 0.95 for moderate and extreme settings) and the less dense models (0.9 and 0.99 for moderate and extreme). While in the moderate setting, the performances similar and in line with the dense model's performance (see Table D in the extreme setting, SparsyFed demonstrated more versatility, achieving high performance even with the sparsest model. It is important to note that in such setting, the model's performance would align with the sparsest model trained, especially considering is trained from a larger (though not majority) group of clients compared to the others.
27

---

Published as a conference paper at ICLR 2025
E.7 WIDE FEDERATED CIFARIO
In this experiment, we aimed to address the challenge of training a model in a setting where the number of samples per client is extremely low. To do this, following an approach similar to the one proposed in Charles et al. (2021a), we partitioned the CIFAR-10 dataset into 1000 clients following an LDA distribution with a 0.1. As shown in Table 5 SparsyFed significantly outperformed the alternative in this setting, where clients retain minimal information. Notably, both sparse methods outperformed the dense one this setting, likely due to the dilution of information in the dense model when such small amount of data is used at each round.
Method
Accuracy (%)
Dense
47.11 2.77
FLASH
51.96 + 2.84
SparsyFed
54.37 1.28
Table 5: Performance comparison of the dense model, FLASH, and SparsyFed. Both FLASH and SparsyFed have been trained with a sparsity of 0.95. The experiments were conducted on CIFAR- 10, partitioned across 1000 clients following LDA with a 0.1, and participation rate of 0. 1..
E.8 VISION TRANSFORMER
To extend our experimental evaluation, we designed an experiment using a Vision Transformer (ViT) (Dosovitskiy et al., (022]) on the CUB-200-2011 dataset (Welinder al., 2010), following the setup proposed in (Hu et al. 2023). Specifically, we use a pre-trained ViT-Base model on ImageNet- 21k (Deng et al., 2009) and fine-tuned on CIFAR- CIFAR-100 The data is partitioned using LDA with 1000.0 among 100 clients, each receiving approximately 60 samples. The participation rate for training and evaluation is set to 10%. The experiments were conducted using the A optimizer with various settings. While hyperparameter tuning can be challenging and resource-intensive in this context, initial results indicate noticeable advantage of using SparsyFed over naive Top-K approach in federated learning. The following results compare four different combinations of learn- ing rate and the number local epochs. In the first case, only one local epoch is performed per warm-up
round, while in the other cases, three local epochs used, with the first serving phase. Additionally, two different learning rate strategies are evaluated: one with a fixed learning rate of 0.01 and another with a server-side decay schedule that reduces the learning rate from 0.01 to 0.001. The global sparsity is set to 50%, and the SparsyFed and Top-K K implementations are compared. Overall. SparsyFed leads dent benefits in some instances. In all settings, SparsyFed can remarkably maintain a sparsity level close the target throughout training. In contrast, the Top-K K implementation tends to drift away from the desired sparsity training progresses. Our method thus reduces communication overhead during training while effectively achieving the target sparsity level in the final model.
consistent improvement in performance, with more evi-
'The pre-trained model sourced the Hugginface Hub from this repository
28

---

Published as a conference paper at ICLR 2025
CUBB-26 21pedam1, Paai (R-123)
CUB--20 (ppederi, Fiad UR-144)
CUS5-31 (bpadbe)), Lit-U)))
Depntontd Lone
Spanato (N
Doining
12
SpancCil Zupt Squiiit
Rounds
Bounds
(1110-000 (ww.aaaccl......... Pladl CZuu 901)
(110-590 (Tpunbac Pial Daut Pialt
CTT 30 Reciand Flaa 12na
Speratt (P)
Disesbeted
Tell Taps ap
OLB-3E Specised Hax (i-010)
(UI1226 Opeeleed
Text
Thittig
Dnsitteted
Spenaty
1g Tape Spurity
Sakk
Sipkk
Thaning Sea 12
Rntertureed Samas
Tec
3.0
151
110
190
26
Figure 13: Experiment using a Vision Transformer (ViT) on the CUB-200-2011 dataset with LDA 1000.0, 100 clients, and 10% partial participation rate for training and evaluation. For each experimental setting (rows), we show the training loss, distributed evaluation loss, and sparsity level on the left, central, and right columns, respectively. The training loss is measured during training. The distributed evaluation loss is collected during the client data evaluation. The sparsity level refers to the global model after aggregation on the server. Different settings are shown, from top bottom: (first row) one local epoch with fixed learning rate, (second row) one local epoch with server-side learning rate decay from 0.01 to 0.001, (third row) three local epochs with a fixed learning rate, and (fourth row) three local epochs with server-side learning rate decay from 0.01 to 0.001. The number of local epochs and the final learning rate are reported at the top of each plot.
29

---

Published as a conference paper at ICLR 2025
E.9
FULL TABLES ACCURACY RESULTS
Dataset
Sparsity
Resnet18
ZeroFL
FLASH
SparsyFed
dense
85.14 + 1.18
0.9
82.16 0.80
78.67 1.52
82.57 + 2.05
84.31 + 0.86
CIFARIO (a = 10%)
0.95
77.92 0.97
76.16 + 1.28
82.22 0.14
84.25 + 1.38
0.99
68.11 = 3.50
72.40 + 1.08
77.48 + 2.86
77.16 + 0.85
0.995
54.49 8.70
60.31 4.11
71.85 = 1.63
72.71 + 0.65
0.999
17.87 = 6.22
21.91 + 1.11
46.43 = 16.73
55.24 + 2.09
dense
53.79 = 1.63
0.9
46.47 1.74
44.11 0.93
53.06 1.10
54.44 1.33
CIFARI00
0.95
13.45 21.45
33.67 + 3.34
54.33 = 1.48
(a = 10%)
0.99
1.54 1.22
10.59 2.48
44.40 1.77
47.62 1.67
0.995
0.97 0.64
4.04 3.43
3.78
42.05
1.21
0,999
0.87 #.21
9.61 3.61
13.85 + 1.01
dense
92.98 0.67
0.9
85.53 0.84
89.12 + 1.11
91.34 + 0.52
Speech Commands
0.95
1.92
85.63 + 1.12
88.16 1.37
89.79 + 0.21
(a 10º)
0.99
67.67 2.32
60.79
76.41
77.00 + 0.73
0.995
36.69 = 1.45
41.16
67.54
70.03 + 1.14
0.999
63.63 3.24
16.77 6.23
31.83 + 2.18
49.27 + 0.50
Table Aggregated results for CIFARIO, CIFARI00, and Google Speech Command datasets.
Dataset
Sparsity
Resnet18
ZeroFL
FLASH
SparsyFed
dense
83.70
1.70
0.9
80.56 1.90
76.16
+ 1.30
81.15 + 1.03
82.13 + 1.53
CIFARIO (a 1.0)
0.95
74.71 + 3.29
75.53
+ 2.27
79.36 + 1.03
82.60 + 1.58
0.99
66.27 5.08
70.71
+ 0.15
73.45 + 1.37
77.71 + 1.69
0,995
63.82 = 2.41
56.02
+ 3.95
69.15 = 1.60
70.01 + 0.43
0.999
31.79 19.10
17.66
+ 8.34
36.07 = 7.49
51.39 + 3.19
dense
52.29 1.14
0.9
46.57 = 1.71
40.70 + 4.72
51.99 -0900.20 0.21
53.08 + 0.90
CIFARI00
0.95
28.07 = 23.27
38.82 + 1.75
47.19 = 1.88
52.81 = 1.72
(a = 1.0)
0.99
19.65 + 16.30
18.97 + 2.08
42.76 + 4.08
46.64 + 1.59
0.995
9.51 + 14.81
6.01 4.74
36.43 - 4.97
42.21 1.03
0.999
3.81 = 2.18
1.96 + 0.66
5.80 + 2.86
15.96 + 0.64
dense
91.49 + 0.94
0.9
84.28 0.88
87.79 + 1.40
88.68 + 1.72
92.32 - 1.59
Speech Commands
0.95
78.58 0.44
84.29 + 1.50
84.89 0.49
89.14 1.15
(a 1.0)
0.99
65.01 - 0.84
+.82
69.22 1.59
75.82 - 3.72
0.995
56.73 1.00
37.16
58.23 = 1.84
68.02 - 3.14
0.999
21.56 = 12.79
10.10 + 4.01
17.70 = 2.58
47.43 - 1.66
Table Aggregated results for CIFARIO, CIFARI00, and Google Speech Command datasets.
30

---

Published as a conference paper at ICLR 2025
Dataset
Sparsity
R
ZeroFL.
FLASH
SparsyFed
dense
73,81 4.84
0.9
69.79 3.78
67.40 + 4.11
71.87 + 2.63
75.00 + 2.78
CIFAR 10
0.95
60.00 4.66
61.55 + 4.18
72.08 + 2.09
75.95 3.39
(a 0.1)
0.99
43.96 = 11.99
51.71
3.54
5
63.69 3.90
0,995
19.02 = 10.77
41.33
+ 3.64
52.15
56.79 + 3,97
0.999
11.5 4.494
18.76
+ 4.28
29.31 6.75
43.68 7.61
dense
48.34 2.71
0.9
41.96 2.16
31.92 +: 7.65
4
48.37 + 1.73
CIFAR100
0.95
11.48
34.21 7.65
48.27 2.70
(a = 0.1)
0.99
0.72
13.07 2.26
34.75 3.38
41.03 + 2.14
0.995
0.14 0.72
7.04 5.25
26.44 = 17.35
35.72
2.01
0.999
0.14 0.72
1.66 + 0.97
3.56 + 2.07
13.84 3.69
dense
80.15 + 2.69
0.9
65.44 0.97
70.35 + 2.65
7
79.67
2.78
Speech Commands
0.95
57.39 = 1.04
65.90 1.88
71.28 1.75
75.46
2.24
(a = 0.1)
0.99
50.42 = 6.26
41.42 1.60
5 2.00
56.69
4.56
0.995
34.20 1.43
22.61 3.45
43.16 3.47
48.30
5.39
0.999
19.25 6.01
8.85 3.76
17.14 + 2.97
29.24 2.34
Table 8: Aggregated results for CIFAR1O, CIFARI00, and Google Speech Command datasets.
ADDITIONAL EXPERIMENTAL CONFIGURATION DETAILS
Learning rate scheduler. The learning rate follows a scheduled pattern defined by the function:
Nend
T) 7start exp
T
In
7/sturt
Reproducibility. Seeds were used for client sampling, while others were fixed for reproducibility purposes. All simulations were conducted using three different client sampling seeds: 5378, 9421, and 2035.
Experimental Setting. Each round consisted of one local epoch with a local batch size of 16 sam- ples. The initial learning rate was set to 0.5, gradually decreasing to a final value of 0.01 following Eq. (T.. For SparsyFed, the exponent for re- ce-paaameterization was set to 3 1.25 for the CIFAR- 10/100 experiments and 3 1.15 for the Speech Commands experiment. The CIFAR experiments were run 700 rounds, while the Speech Commands experiment was run 500 rounds.
G COMPARATIVE ANALYSIS ON ALGORITHMS AND BASELINES
Here is brief description of the implementation used during the experiments for Top K, ZeroFL
Qiu al. 2022). FLASH Babakniya et al. (2023), and SparsyFed:
G.1 TOP-K
1. Pruning: The model is pruned per round at clients after executing the (local) training on their own data, i.c., just before sending the updated model to the server. The pruning method used is global unstructured Top-K, which prunes all the model parameters except for the i largest values.
2. Aggregation strategy: FedAvg. Alternatively, other aggregation strategies acting on the pseudo-gradiens gradients can also applied straightforwardly.
31

---

Published as a conference paper at ICLR 2025
G.2 ZEROFL
Pruning is performed during the local training at clients and before sending the model update back to the central server. During training, the SWAT unstructured (per- layer, contrast to standard global unstructured Top-K) approach is used to prune the weights before the forward pass and the activations during the forward pass. This alone doesn't t ensure obtaining a model with the targeted degree of sparsity because SWAT often results in weight regrowth per optimizer step. To achieve the target sparsity, the model is pruned again before sending it back to the server, using the same approach as Top K, i.e., global unstructured. proposed to increase the density of the model before the uplink communication. For our experiments, we used the smaller one (0.0) since is more in line with
Pruning:
the original work, three levels of masks (00.0, 0.1, 0.2)) have been
2. Aggregation strategy: A slight variation of FedAvg is used, where the averaging is exe- cuted among the non-zero weights to avoid excessive dilution of information in the pres- ence of highly sparse models. Adapting the aggregation function to act only non-zero weights supports alternative aggregation strategies acting on the pseudo-gradients.
G.3 FLASH SPDST (SENSITIVITY-DRIVEN) PRE-DEFINED SPARSE TRAINING)
(THE ONE USED IN THE EXPERIMENTS)
1. Pruning: Pruning is performed at the end of the first round of training (similar to Top- rop-K), which, in the original paper, referred as warm-up phase. Thus, the first round of training is executed using the dense model, producing the initial mask. The binary mask obtained during the warm-up phase is fixed for the subsequent federated rounds, and only the non-zero weights are trained. This means there is no need for further pruning of the weights in subsequent rounds since no regrowth allowed (in this version of Flash).
2. Aggregation strategy: The aggregation is performed only among the non-zero weights, similar to ZeroFL. During the first aggregation, at the end of the initial training round, the model is further pruned on per-layer basis to counter the regained density caused by the mismatch local masks. The pruning uses the average sparsity level all clients achieve for each layer. For example, if the average sparsity of layer 7 is di, adjusted by factor r, then layer I of the pruned global model will have sparsity d. The factor r helps to maintain the target global sparsity by ensuring that the sum of individual layer sparsity levels meets the overall goal. The resulting binary mask becomes the final one, preserved for all subsequent training rounds. This process, defined in the original paper as sensitivity analysis, is applied only in the first round, as fixed mask training prevents further mask modification in later rounds.
G.4 FLASH JMWST (JOINT MASK WEIGITT SPARSE TRAINING)
1. Pruning: A normal training procedure is applied (NO FIXED MASKS). Pruning is per- formed at the end of each local training session.
2. Aggregation strategy: The server aggregates and then prunes the model, applying the same sensitivity analysis introduced in SPDST to address the regained density. This is done every round, as the training method allows for the regrowth of the clients. The original paper proposed two values for r: and r = 5.
G.5 SparsyFed
1. Pruning: Powerpropagation is applied to re-parameterize the weights during the forward pass executed at clients. Pruning during training applied only to the activations during the backward pass. At the end of the local training, the model is pruned and returned to the server. The first round of training executes using a full-size model.
2. Aggregation strategy: FedAvg. Alternatively, other aggregation strategies acting on the pseudd-gradients can also be applied straightforwardly.
32

---

Published as a conference paper at ICLR 2025
H FLOPS REDUCTIONS FOR UNSTRUCTURED SPARSE TRAINING.
Unstructured sparsity can theoretically reduce FLOPs linearly in the percentage of zero weights (Dettmers & Zettlemoyer 2019 Singh al. 2024) an optimal sparse algorithm exists for a given operation. However, such gains are fully realized only when the hardware can effi- ciently skip pero-valuee operations. In contrast, structured sparsity methods- -ucch as block-sparsity pruning (Gray et al., 2017) - yield more predictable speed-ups on standard GPUs due to optimized kernels despite typically achieving lower overall sparsity than unstructured sparsity. Emerging ac- celerators and libraries increasingly support unstructured sparsity (Jeong et al. 2024). bridging the gap between theoretical FL.OP reductions and actual runtime improvements.
33


Reviews:
[{"review_id": "aJfs0VhJXj", "reviewer": "Reviewer_yVEF", "summary": "This paper discusses utilizing sparse learning to reduce communication and computation costs in cross-device federated learning. The authors propose the SparsyFed algorithm, where in every round, the participants first reparameterize the weights and then perform the forward pass. In the next step, before calculating the gradient, each client prunes the activations in each layer. Finally, it only sends back the TopK large gradients to the server. This way, they can reduce client communication costs to the server.", "questions": "* Why did not the authors not consider other existing works they mention as their baseline?\n\n* Could the authors explain the distinction between their paper and a naive federated version of [Schwarz et al. (2021)]? \n\n* Is the sensitivity of this method to client participation rate, number of clients, and number of data points within each client? Is it possible that when some clients do not have enough data, SparsyFed might fail to converge or find a proper sparse model?\n\n* What is the computation cost of steps 8 - 9 - 10 in algorithm 2 compared to full training?\n\n* In the appendix, the authors presented two versions of the FLASH algorithm, one of which does not fix the mask (G.4). Could the authors explain why they selected G.3 over G4? Because it seems that G4 does not have the plasticity problem.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The writing, motivation, and related works have been clearly explained.\n\n* The algorithm achieves high sparsity ratios and communication cost reduction with low-performance degradations.\n\n* The authors have presented interesting results in the evaluation section.", "weaknesses": "* The novelty of the algorithm is not apparent.\n\n* While the paper is well-written, and the authors have successfully conveyed the problem statement, the algorithm is not clearly explained. For example, how does step 8 work?\n\n* It is not clear why different decisions have been made and what effect each of them has on the final performance. For example, why does weight reparameterization work, or how does layer-wise top-k selection result in layers with different sparsities? \n\n* The experiment section only covers resent18, and it is unclear if this method works on other types or sizes of models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0qMwaiGnx6", "reviewer": "Reviewer_wQKr", "summary": "SparsyFed is designed for cross-device environments where devices have limited computational and communication resources. Traditional federated learning approaches struggle with communication overhead, computational demands, and data heterogeneity across devices, especially with dense models. SparsyFed addresses these challenges by introducing sparse adaptive federated training, which reduces the model’s memory and computational requirements while retaining high accuracy. The key innovations include a sparsity-inducing weight reparametrization technique that requires only a single additional hyperparameter and a method for pruning activations to reduce computational load on clients. SparsyFed achieves up to 95% sparsity with minimal accuracy loss, which is particularly beneficial for constrained edge devices in cross-device FL.  SparsyFed demonstrates significant improvements over traditional and recent sparse FL baselines, achieving a 19.29× reduction in communication costs and faster global model convergence. The challenges of data heterogeneity, limited client resources, and the need for high sparsity without complex hyperparameter tuning are addressed through its adaptive and lightweight design.", "questions": "Asked in weakness", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well written and has a solid motivation. The strength of this paper is as follows,\n\n1. SparsyFed can achieve target sparsity levels of up to 95% while maintaining competitive accuracy. This is particularly beneficial in federated learning environments, where models often face challenges from heterogeneous data distributions​.\n\n2. The method significantly reduces communication overhead, achieving up to 19.29 times less communication than dense models. \n\n3. This is crucial in cross-device federated learning where devices often have limited bandwidth unlike many federated learning methods that require multiple hyperparameters, SparsyFed simplifies the tuning process by needing only one hyperparameter.", "weaknesses": "This paper is experimentally solid but theoretically weak. The weaknesses of this paper are as follows,\n\n1.\tThe authors mentioned the model’s sparsity by reducing the FLOPs and memory footprint in lines 207 and 208. However, the authors have not given any experimental evaluation of it. It would be good to give a FLOP analysis of SparsyFed with the dense model. \n\n2. In experiments (line 325) “19.29×less communication costs compared to the dense model” what is the convergence bound of the proposed algorithm? Is it similar to FedAvg? Why is SparsyFed more efficient in communication than state-of-the-art?\n\n3. I do not understand Figure 1 (right). Is it the comparison of the different local models or the global model? What happens when the target sparsity is more than 90%? I think author should elaborate more on  Consensus on the spare masks across clients (Section 5.3).\n\n4. In Table 2, If we increase sparsity why SparsyFed with activation pruning perform worse in both IID and non-IID scenarios? What happens when $\\alpha$ =0.1 ? The inference from the observation is not clear to me. \n\n5. In Figure 3, the authors proposed these reparameterization methods or those available in the state-of-the-art ? What authors tried to show here is unclear to me could author please elaborate more on Figure 3 ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YIRyIeSaIY", "reviewer": "Reviewer_LGTa", "summary": "SparsyFed introduces a novel approach to federated sparse training that achieves high model sparsity (95%) while maintaining accuracy. To create a practical federated sparse training method that addresses the issues of data heterogeneity, adaptability, and hyperparameter tuning, SparsyFed produces highly sparse models with negligible accuracy degradation, requires only one hyperparameter, achieves significant weight regrowth reduction, and maintains model plasticity under sparse design.", "questions": "see weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Only needs one hyperparameter (sparsity target), making it very practical for FL settings where tuning is challenging\n2. Maintains accuracy up to 95% sparsity, significantly outperforming baselines\n3. Good ablation studies on weight reparameterization and activation pruning", "weaknesses": "1. The explanation would be clearer if a figure illustrating the pipeline of the proposed method were provided.\n2. This reviewer believes that focusing on fewer hyperparameter tuning efforts should not be considered the main contribution (or even the primary motivation), especially since addressing this challenge isn't the primary focus of the work. Furthermore, while the proposed method introduces only one hyperparameter, existing hyperparameter-related issues in FedOpt still persist.\n3. The reviewer finds it confusing that Line 10 in Algorithm 2 appears redundant given the presence of Line 9.\n4. Is sparse communication essential for the proposed method to address the three challenges it focuses on?\n5. The paper lacks sufficient discussion on why the proposed method facilitates more effective consensus on the sparse pattern.\n6. There is an absence of theoretical analysis or guarantees regarding the convergence properties of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7CgQL3B8YU", "reviewer": "Reviewer_Ay19", "summary": "This work presents SparsyFed, an adaptive sparse training method tailored for cross-device federated learning. Authors show that SparsyFed can achieve impressive sparsity levels while minimizing the accuracy drop due to the compression. SparsyFed outperforms in accuracy three federated sparse training baselines, TopK, ZeroFL, and FLASH, using adaptive and fixed sparsity for three typical datasets used in cross-device FL.", "questions": "See weaknesses.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.The authors summarize three challenges when process sparse training in cross-device federated learning environments.\n\n2.The authors propose the SparsyFed method to accelerate on-device training in cross-device FL.\n\n3.Extensive experiments show the effectiveness of SparsyFed, which can target high sparsity levels (up to 95%) without compromising the model’s accuracy with a novel approach leveraging hyperparameter-less local activation pruning and thoughtful weight reparameterization.", "weaknesses": "1.It’s suggested that authors give a comprehensive survey on adaptive sparse training method. Although authors claim “Previous works have only managed to solve one, or perhaps two of these challenges”, can authors give a comprehensive comparison of existing methods? \n\n2.Considering different clients train different submodels, the server also maintains a full model. So can the sparsity of clients be different to apply for heterogeneous hardware?\n\n3.Can authors further explain why clients should achieves consensus on the clients’ sparse model masks when server always maintain a full model.\n\n4.What’s the definition of the model plasticity?\n\n5.In experimental section, authors only compared with two baselines, there’re several works also focus on the same questions, for example [1,2,3], so it’s suggested to add more experimental to show the effectiveness of proposed method.\n\n6.Considering the model architecture, authors only show the effectiveness on convolutional network, what’s the performance on other architecture, for example Transformer?\n\n[1]Stripelis, Dimitris, et al. \"Federated progressive sparsification (purge, merge, tune)+.\" arXiv preprint arXiv:2204.12430 (2022).\n\n[2]Wang, Yangyang, et al. \"Theoretical convergence guaranteed resource-adaptive federated learning with mixed heterogeneity.\" Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023.\n\n[3]Zhou, Hanhan, et al. \"Every parameter matters: Ensuring the convergence of federated learning with dynamic heterogeneous models reduction.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cphHV2urP8", "reviewer": "Reviewer_75sH", "summary": "This paper proposes a sparse FL method to tackle the following problems in FL: 1) data heterogeneity and its impacts on sparsity across clients; 2) adapt to never-seen data distributions; 3) hyperparameter turning in FL. The authors present *SparsyFed* and claim that it can produce very sparse models, while offering *plasticity* to data distributions.", "questions": "See the comments above.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors conduct comprehensive empirical evaluations of the proposed methods. In particular, the proposed method is tested on multiple datasets with different data distributions. Besides model accuracy, the authors also analyze reductions in communication costs when using the sparse training method. \n\nIn addition, the authors also conduct necessary ablation studies to investigate 1) sparsity pattern dynamics during training; 2) different weight re-parametrization methods; 3) activation pruning.", "weaknesses": "**1 Claims are not fully supported**: The authors claim that *SparsyFed* addresses all three problems mentioned in the abstract: data heterogeneity and its impact on clients' sparsity, a lack of sufficient plasticity to never-seen data distributions, and hyperparameter tuning. However, it is unclear that what techniques in *SparsyFed* address these problems. In particular, it is not convincing that consensus in sparsity is achieved by simply adopting a weight re-parameterization scheme. I believe more analyses are needed to support the claim.  \n\nIn addition, I am also confused about how *SparsyFed* can `adapt to never-seen data distributions`, as there is no technique used in the method that can specifically address this problem.\n\n**2 Presentations in the introduction section**: I did not see any part in the introduction explaining technical details in *SparsyFed*. The authors mainly focus on describing the benefits of *SparsyFed*. However, it is confusing what techniques in *SparsyFed* bring these benefits.\n\n**3 Contributions are not fully explained in Sec 3**: \n\nFirst, I am confused about lines 6-10 in Algo 2. Some functions like *GetLayer* and *SetLayer* are not explained. As these lines are the main contribution to the paper, it is hard to understand why *SparsyFed* achieves good performance without knowing its technical details. \n\nFurthermore, the authors did not explain why sparse activations help attain a sparse model. As it is not a theoretical work, more intuitive explanations and evaluations are needed to justify such operations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)