SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learning
Authors: Riyasat Ohib
Bishal Thapaliya
Gintare Karolina Dziugaite
Jingyu Liu
Vince D. Calhoun
Sergey Plis
Venue: Accepted by TMLR ()

Abstract: In this work, we propose Salient Sparse Federated Learning (SSFL), a streamlined approach for sparse federated learning with efficient communication. SSFL identifies a sparse subnetwork prior to training, leveraging parameter saliency scores computed separately on local client data in non-IID scenarios, and then aggregated, to determine a global mask. Only the sparse model weights are trained and communicated each round between the clients and the server. On standard benchmarks including CIFAR-10, CIFAR-100, and Tiny-ImageNet, SSFL consistently improves the accuracy–sparsity trade-off, achieving more than 20\% relative error reduction on CIFAR-10 compared to the strongest sparse baseline, while reducing communication costs by $2 \times$ relative to dense FL. Finally, in a real-world federated learning deployment, SSFL delivers over $2.3 \times$ faster communication time, underscoring its practical efficiency.

Published in Transactions on Machine Learning Research (12/2025)
SSFL: Discovering Sparse Unified Subnetworks at Initializa- tion for Efficient Federated Learning
Riyasat Ohibi*
Shyasat.ohii&gaalee edu
Georgia Institate of Technology
Bishal Thapaliyal*
bthepaakyo.1@@@gaal.ooo com
TReNDS Center
Gintare Karolina Dziugaite
gkdz
Google DeepMind, Mila Quebec AT Institude
Jingyu Liu
35su75 @gsu. edu
Georyia State Universily
Vince D. Calhoun
realhown@@gteechccoo edu
Georgia State University, Georgia Instttttte of Technology, Emory University
Sergey Plis" Georyia State Universily
wm..pis@@gmall.omm com
Center for Translational Research in Newrodnaging & Dala Science (TReNDS), Georyia State Universily, Georgia Institute of Technology, and Emory Universily.
Reviewed on OpenReview: https: // openreview net/ net/formmm id= kUZ6LAUB26
Abstract
In this work, we propose Salient Sparse Federated Learning (SSFL), streamlined approach for sparse lederated learning with efficient communication.. SSFL identifies sparse subnetwork prior to training, leveraging parameter saliency scores computed separately on local client data in non-IID scenarios, and then aggregated, to determine a global mask. Only the sparse model weights are trained and communicated each round between the clients and the server. standard benchmarks including CTFAR-10, CIFAR-100, and Tiny- NetgeNt,,,, SSFL consistently improves the accuracy sparsity trade-off, achieving more than 20% relative error reduction CIFAR-10 compared to the strongest sparse bascline, while reducing communication costs by 2x relative to dense FL. Finally, in real-world federated learning deployment, SSFL delivers over 2.3x faster communication time, underscoring its practical efficiency. Our code is available at: attps://g www.tt....o.riihiiti coab//oooo/////SP.
Federated Learning, Sparse Neural Networks, Model Pruning, Pruning at Initialization, Sparse Subspaces, Communication Efficiency, Non-IID Data.
Keywords:
1 Introduction
The success of deep learning has been propelled by ever-larger models trained on centralized datasets. Yet, in critical domains such as healthcare, finance, and mobile computing, data inherently decentralized, privacy-sonsitive, and distributed across millions user devices or institutional silos. Foderated Learning (FL)
"These authors share first authorship. Correspondence riyasat.
riyanat.ohibbgateel. wwwb@gatech.....momoooooo edu; brhapal.ya16@@mmil.ccmm

---

Published in Transactions on Machine Learning Research (12/2025)
has emerged as compelling paradigm for such settings, enabling collaborative model training without direct data sharing (McMahan & Ramage, 2017). By orchestrating learning across clients who retain their local data, FL promises privacy preservation and data locality. However, practical deployment of FL, especially in cross-device scenarios, remains fraught with challenges.
FL. systems must contend with severe system heterogeneity, where client devices exhibit widely varying compute, memory, and energy resources. This is compounded by statistical heterogeneity, as data across clients is typically non-IID and differently balanced, making it difficult to train single global model that generalizes well across users (Kairouz et al,, 2021). These challenges often manifest communication bottlenccks, poor convergence behavior, and brittle hyperparameter tuning (Khodak et al., 2021).
One promising approach to alleviating these boltkss is through sparse training, which reduces both the on-device computation and the communication payload (Lin et al., 2017; Wang et al., 2020; Evci et al., 2020a). Among sparse FL methods, there has been considcrable progress dynamic sparsification strategies that evolve sparse masks over the course of training (Bibikar al., 2022; Dai et al., 2022; Guastella et al., 2025). However, these methods typically rely on iterative coordination, complex heuristics such prune-and-grow schedules, and introduce additional hyperparameters that require tuning. Moreover, certain pruning-at-iniiee itaiialattinttiilizzttoon (Pal) approaches resort to using public proxy datasets to identify subnetworks, which violates the privacy-preserving ethos of FL (Huang et al., 2022). Beyond operat operational ional complexity, dynamic masking approaches may suffer from a more fundamental challenge: as masks evolve independently across clients, parameter updates occur in shifting subspaces, potentially hindering the formation of coherent global representations.
In this work, we address fundamental open problem in sparse and efficient FL: how performant, static sparse subnetwork at initialization using only private, non-IID data across clients with minimal communication and no auxiliary dataset? To this end, we propose Salient Sparse Federated Learning (SSFL), a simple yet effective method that discovers a globally shared sparse mask prior to training via a single step of distributed saliency aggregation at the start of the learning process. Each client computes local gradient-based parameter importance scores on private data and transmits them once to the server, which averages them in a protoorrpotti manner to construct a global heatmap of saliency scores. The Top-k operation on the aggregated saliency heatmap defines fixed binary mask, establishing a common sparse surnetwork for all clients that is preserved for the duration of training. By constraining the federation to this unified sparse topology, we ensure optimization proceeds within a consistent parameter subspace, avoiding the potential instability of shifting masks found in dynamic approaches. This leads to a highly ceanuunicatiio-fffciiree and initial mask discovery.
identify
static sparse FL pipeline, with zero overhead beyond the
Extensive experiments across CIFAR-10, CIFAR-100, and Tiny-ImageNet under realistic non-lII settings demonstrate that SSFL significantly outperforms both dense and sparse FL baselines, including state-of state-of-the-ar dynamic sparsity methods such as DisPFI, (Dai et al., 2022), SparsyFed (Guastella et al., 2025) and Flash (Babakniya et al., 2023). Morcover, we validate the practical efficiency of SSFL in real-world FL deployment, achieving over 2.3x wall-clock communication speedups on larger models. To the best our know knowledge, SSFL is the first method to offer a single-shot, fully decentralized, and privacy- preserving sparse FL strategy with competitive performance across standard benchmarks.
Contributions
We summarize our contributions as follows:
We introduce SSFL, the first single-shot sparse federated learning framework that discovers shared sparse subnetwork at initialization using only local client data. Our approach avoids the need for public auxiliary datasets, iterative pruning cycles and additional hyperparameters.
a globally
We develop a commuicatiinn-ffiieen mechanism to compute globally representative importance scores by aggregating local saliency scores in a single round of coordination. This aggregation respects both data heterogeneity and client privacy, providing a practical solution for stable sparsity in foderated learning.
2

---

Published in Transactions on Machine Learning Research (12/2025)
We demonstrate that SSFL consistently outperforms state-of-the-art sparse federated learning baselines, including recent. dynamic methods such as DisPFT. and SparsyFed (Section 4.1). On CIFAR-10, SSFL, achieves over 20% relative error reduction compared to the strongest sparse baseline, and it delivers similarly strong accuracy-sparsity improvements across CIFAR-100, Tiny-Imageeet, and diverse non-IID partitions. On ResNet-50, the performance gap widens over DisPFL, from +23% at 50% sparsity to +35% at 95% sparsity, where DisPFL collapses to 12% while SSFL maintains around 48% accuracy.
We validate SSFL's effectiveness through ablation studies showing: ! SSFL approaches oracle salient subspace quality with increasing client participation in Section 4.2.3 @ the discovered subspaces encode meaningful structure, as evidenced by performance degradation under permutation in Section 4.2.2 the framework can adapt to out-of-distribution data when needed in Section 4.3.
We deploy SSFL in a real-world federated learning system and observe over 2x faster wall-clock communication compared to dense baselines (Section 4.1.1), underscoring the method's practical impact.
2
Related Work
Our work lies at the intersection of model pruning and communication--ffieee federated learning (FL). We group related efforts based on how they impose sparsity and highlight how SSFL enables one-shot global sparse training using only private, non-IID data and single round of communication.
Dynamic sparsity in federated learning. A growing body of work explores dynamic sparsity, where sparse masks evolve during training. These methods are designed to adapt to changing data distributions but typically require repeated coordination between clients and the server. For example, SparsyFed (Guastella et al., 2025) combines activation pruning with adaplive reparameterization, while DisPFL (Dai et al., 2022) and FedDST (Bibikar et al., 2022) adopt the RigT. strategy (Evei et al., 2020a), periodically regrowing pruned weights during training. DSFL (Beitollahi et al., 2022) introduces user-adaptive compression rates, MADS (Yan et al., 2025) proposes mobility- aware sparsification that adjusts based on contact time and model staleness, and DSFedCon (Li et al., 2025) integrates dynamic sparse training with federated contrastive learning.
Although effective, such methods introduce additional hyperparameters and require multiple rounds of communication for mask updates. Moreover, because clients maintain different and evolving masks, the server must aggregate updates defined over partially non-overlapping parameter subspaces, complicating aggregation and oflen leading to denser effective global models. contrast, SSFL discovers a single global mask at initialization, ensuring that all updates are aligned in the same sparse subspace and avoiding iterative coordination altogether. This also provides more stable optimization trajectory, clients remain in common subspace throughout training.
Structured sparsity and specialization. Some FL methods impose structured sparsity by pruning filters, blocks, layers reduce memory and inference costs constrained devices. HeteroFL (Diao et al., 2021) assigns clients submodels based on compute budgets, while AdaptCT. (Zhou et al., 2021) employs adaptive structured pruning to synchronize FL processes across heterogeneous environments. Sub-FedAvg (Vahidian et al., 2021) combines both structured and unstructured pruming for personalized federated learning under data heterogeneity. More recent work includes subMFL, (Oz et al., 2024), which generates compatible submodels through server-side structured pruning, and AdaPruncFL (Fan al., 2024), which proposes data-free adaptive structured pruning for heterogeneous client capabilities.
While structured sparsity primarily targets computational efficiency, the dominant bottleneck in federated learning lies in communication. Unstructured sparsity better addresses this by maximizing parameter reduction without. hardware-speccffic constraints. Importantly, the sparse local models produced by SSFL can still benefit from mustructured-spariity aware accelerators (e.g., SPMM kernels), enabling device-level specdups(Nakahan et al., 2019; Thangarasa et al., 2023; Gale et al., 2020; NVIDIA, 2021; 2020a). Thus, whereas structured approaches emphasize client specialization and hardware compatibility, SSFL focuses on global communication savings while remaining compatible with hardware acceleration.
3

---

Published in Transactions on Machine Learning Research (12/2025)
C
Ci
P1 81
Wi e
77
2
= Σ 8kPk
Top
C
8282
Biuary Mask
@ m
C
C
Wk
Mr O
Dense Local Models
saliency scores
Sgarso Models
With Commoo Manlks
Illustration of the distributed comnection importance in the non-lID setting. The parameter saliency scores from each site calculated on local minibatches equal class distribution are aggregated, weighing them with the proportion of the data available that site. The common mask generated from that score is applied local client models.
Figure I:
Static sparsity and pruning gruning--t-iniiie aatiinitialization in FL. Pruming-at-initialn at-initialization (Pal) techniques aim to identify a sparse subnetwork before training begins. Although well-studied in centralized training (Lee et al., 2018; Wang et al., 2020; Tanaka et al., 2020), their application in federated learning remains limited. Several methods circumvent the challenge by relying on public proxy datasets to estimate parameter importance, as in FedTiny (Huang al., 2022). Others compute masks locally but struggle generalize across clients under non-IID distributions (Jiang al., 2022). FedPal (Wang al., 2025) proposes progrossive, multi-round praning strategy, but this introduces additional pruning stages and communication costs. SSFL addresses this gap by introducing single-shot, privacy-preserviie using only local client data. To the best of our knowledge, it is the first to demonstrate that gradient-based saliency scores, when aggregated keeping the data disparity mind, can enable static sparse training FL without auxiliary public datasets or iterative coordination during the training process.
approach to identify high-quality global mask
A more detailed survey of related pruning and sparse federated learning approaches, including additional variants and historical context, is provided in Appendix B.
3 The Salient Sparse Federated Learning (SSFL) Method
Federated learning poses unique challenges for sparsity: decentralized data, non-IID samples, and limited commmication make it difficult to identify globally effective sparse subnetworks. In particular, pruning-at- initialization (PaI) strategios, while effective in centralized settings, require careful adaptation to function under these constraints. SSFL addresses this challenge by introducing single-shol, communcattion-eliieenn method that constructs a shared sparse subnetwork before training begins. The approach requires only a single round of client-sers server interaction, using local saliency scores to discover a globally important subset of parameters. Once the mask is generated, all further training proceeds on the same sparse subnetwork using standard federated averaging.
4

---

Published in Transactions on Machine Learning Research (12/2025)
Our method consists of two main phases: (i) one-time sparse mask discovery phase based distributed gradient saliency aggregation, and (ii) sparse training phase where clients perform local updates and global aggregation restricted to the space of parameters selected through the mask.
3.1 Parameter Saliency at Initialization
We begin by defining the importance criterion used to identify salient parameters prior to training. Saliency- based pruning methods estimate how much each parameter contributes to the loss at initialization. This idea, initially proposed in early neural network pruning work (Mozer & Smolensky, 1988) and refined in methods like SNIP (Lee et al., 2018), contral to SSFL's mask discovery phase.
Given a randomly initialized model wp E Rd and loss function L(w) evaluated on dataset D, the saliency score sj for parameter wy is defined as:
aC(wo: D) Omy
Wij
(1)
This score captures the first-order sensitivity of the loss to removing parameter wy initialization. Parameters with larger magnitudes of sj are considered more important and are prioritized for retention.
In SSFL, each client computes this saliency score locally Because this estimation is done only once, at initialization, the method incurs additional runtime overhead during training. The main challenge lies in extending this centralized scoring mechanism to federated setting in a way that respects data heterogeneity and avoids repeated communication.
minibatch sampled from its private data.
3.2 The SSFL Process: Single-Shot Mask Discovery and Sparse Training
SSFL extends the saliency criterion from centralized training to federated settings through a single-shot distributed process. This involves computing local importance scores at each client, aggregating them in a datg-weighted manner to produce a global importance voctor, and selecting the top-scoring parameters to define a shared sparse subnetwork. The full procedure consists four stages:
Local Saliency Estimation
At initialization, all clients synchronize to a common model we E Rd.. Each client h samples minibatch By C Dx of size B from its private data and computes the local empirical loss for the obtained batch
L(wo; Ba)
1
Σ
(wwn; xr, y)-
(2)
B
(m.
Using this loss, client i computes local saliency scores sy E Rdd by applying the importance critcrion Equation (1) its minibatch:
aL(wwo B&)
Sk.j
wo.j
Z
This process is fully local and requires no communication. We use only single minibatch client to minimize computation and preserve privacy. A detailed analysis of the saliency estimation is provided Appendix A.1.
Aggregation Saliency Scores.
Each client transmits its saliency scores 8y and local dataset size n& [Dxl to the server. A global saliency vector $ E IRd is then computed via weighted average:
K
where
Ph
(3)
Σ i=1
5

---

Published in Transactions on Machine Learning Research (12/2025)
This data-weighhg weighted aggregation ensures that clients with more examples exert proportionally greater influence the final importance scores, which helps mitigate heterogeneity bias. The class-balanced minibatch sampling further ensures that the computed saliency scores are representative of all classes available in the local clients.
Global Mask Selection.
Given a target sparsity level a E (0, 1), the number of active parameters is & = [(1 ad) o) dj.. A binary mask m E {0, 1] is then computed as:
m = (s, R):
where (opK( f)) returns a binary vector with ones the indices of the h largest values in This mask is broadcast to all clients and remains fixed throughout training.
Sparse Federated Training.
Training proceeds in standard rounds of local updates and model aggregation, with all operations restricted to the masked subnetwork. At communication round r, each participating client & receives the global sparse model w.m. and performs T local update steps.
During training, both forward and backward propagation operate exclusively the sparse subnetwork defined by the mask m. At local step the forward pass computes the loss using only the active (non-zero) parameters:
L(w @ m; Bx.)).
where wi.m m ensures that maskod-out parameters contribute zero the network computation. The corresponding masked gradient update then applied as:
++1 w n (V WWwC(wk m: BR.,) m) kkm
where BR is local minibatch. This ensures that gradients are computed and applied only for the active parameters, while masked parameters remain fixed at zero throughout training.
After T steps, each client sends its sparse model back to the server, and aggregation proceeds as in standard FedAvg. Because the mask is fixed, all communication and computation can be compressed accordingly using standard sparse encodings (e.g., CSR, COO, bitmask formats); we discuss the communication accounting protocol in detail in the Section 4.
3.3 Dataset and non-IID partition
We evaluate SSFL on CIFAR-10, CIFAR-100 Krizhevsky al. (2009) and the TinyImagenet dataset. For simulating non-identical data distributions across the federating clients we use two separate data partition strategies. We next present these non-lID data partition generating strategies and provide a more detailed treatment in Appendix A.2.
Dirichlet Partition We use the Dirichlet distribution to simulate non-llD distribution of data among the clients Hsu et al. (2019) and call this the Dirichlet Partition our experiments. We assume that each client selects training examples independently, with class labels distributed across N classes according to a catogorical distribution defined by the vector q, where cach qi 2 for E (1,2,3...,NN and ||q]] = 1.
To model diverse set of clients, each with distinct data, draw q Dir(ap) from the Dirichlet distribution to determine the class distribution vector q. Here, the vector establishes the baseline distribution across the N classes, while the concentration parameter a > O, dictates the degree of variation or the property of IID in class distribution among clients. By adjusting a, we can simulate client populations with varying degrees of data uniformity. Specifically, as a - oo. the class distributions of the clients converge to the baseline prior distribution pr converscly, as is chosen to be smaller, incrcasing fewer classes chosen random dominates the proportion of data at each class, representing a higher degree of non-IID distribution. As a 0, each client holds data from particular class chosen at random.

---

Published in Transactions on Machine Learning Research (12/2025)
Algorithm 1 SSFL: One-shot sparse subnetwork discovery and training
Require: Number of clients K, communication rounds R, local update steps T, sparsity level
Ensure: Final sparse global model w
Phase 1: One-Time Mask Generation
1: Server initializes model wo and broadcasts to all clients.
for all clients & (l...., ......K } in parallel do
Sample minibatch Bg C D
Compute local saliency vector 8ę using equation 1 |Dx| to the server
Send and local data size
6: end for
7: Server computes data proportions n2k E 1 74
Server aggregates global saliency: Σk=1 Pk.
Server sets number of active parameters & [(1 σ) .d]
10: Server generates global mask m + (op R.)
11: Server broadcasts mask m to all clients
12: Server initializes sparse global model: w g.m + wy 3 m
Phase Sparse Federated Training
13: for r to R do
14: Server selects subset of clients C' S (1....,KK K}
15: Server sends current sparse global model wg.. to all clients in C'
16:
for all clients & C' in parallel do Initialize local model wy < = for t= to
17:
18:
do Sample minibatch ВR, C DD Compute masked gradient: Vi(ww; Bx,,) m
19:
20:
21:
Update local model: wk wk VV m end for
22:
23: Send updated local model wr to server
24: end forServer aggregates client models: w + ΣkEr
ng.
Wk
sea By
25:
end for
In this work, we partition the training data according to a Dirichlet distribution Dir(a) for each client and generate the corresponding test data for each client following the same distribution. We set the prior distribution p be uniform over N classes and specify the 0.3 for CIFAR10 and a = 0.2 for the CIFAR-100 for fair comparison to previous work (Dai al., 2022; Bibikar al., 2022). A detailed description of the Dirichlet partitioning scheme with figure depicting client data allocation is provided in Appendix A.2.
Pathological Partition Following prior work on non-lII federated learning settings (MeMahan al., 2017; Zhang al., 2020), we simulate pathological data partitions, where each client randomly assigned only limited number of classes from the total number of classes. In particular, each client receives at most 2 classes for CIFAR-10 and 10 classes for CIFAR-100.
4 Experiments
We evaluate SSFL against broad range of federated learning methods to demonstrate its effectiveness. Our experimental design is four-fold: (1) We first compare SSFL with statc-of-the-art sparse and dense FL baselines on non-IID distributions of CIFAR-10, CIFAR-100, and ImayeeageNe.. (2) We then study the performance of SSFL under varying sparsity levels up to 95% and compare it to other sparse FL methods. (3) We deploy our method real-world FL framework report wall-clock time improvements. (4) Finally, we analyze the structural propertics of the discovered sparse mask by studying the effect of intra-layer permutations and the quality of the generated mask.

---

Published in Transactions on Machine Learning Research (12/2025)
4.1
Main Results
Our primary results are organized by the data partitioning strategy to clearly delineate performance under different non-lID conditions. All experiments in this subsection use ResNet18 architecture with 50% sparsity.
4.1.1 Performance on varying dataset and partitioning schemes
Performance Dirichlet Partition. Table shows the performance of all methods under the Dirichlet non-IID data partition. On CIFAR-10, SSFL achieves the highest accuracy (88.29%) among all sparse methods and also outperforms the strongest dense baseline, FedAvg-FTT (88.02%). On CIFAR-100, SSFL reaches 61.37% accuracy, again surpassing all sparse and dense techniques. This corresponds to more than a 20% relative error reduction compared to the strongest sparse baseline (DisPFL), underscoring the substantial margin by which SSFL improves over prior work. These results demonstrate that the global mask discoverod by SSFL effectively captures salient features that generalize well across clients with heterogeneous data distributions. Results averaged over five runs.
Performance on Pathological Partition. We further stress-test our method using a pathological non-IID partition, where each client holds data from a very limited number of classes. The results, shown in Table 2, highlight SSFL's robustness. SSFL achieves the highest accuracy on CIFAR-10 with 94.61%. On CIFAR-100, its performance (52.01%) is highly competitive with the top-performing dense baseline, FedAvg-FT (52.47%), while significantly exceeding other sparse methods like DisPFL (44.74%). This indicates that SSFL's saliency aggregation mechanism can construct high-quality global subnetwork even when clients have extremely skewed, non-overlapping data distributions.
Performance on TinyImageNet. To evaluate scalability, we conducted experiments on the more chal- lenging Tiny-ImageNet dalaset. As detailed in Table 2, SSFL again achieves the best performance with 19.4% accuracy, outperforming both dense and sparse counterparts. For this dataset we report results for the strongest baselines from our CIFAR-10/ /000 experiments. These result confirm that SSFL's data-driven subnetwork discovery effective not just on smaller datasets but also scales successfully to more complex image classification lasks.
Experiments the larger ResNet-50 architecture To demonstrate the scalability of SSFL beyond standard benchmarks, we conduct experiments using ResNet-50 on CIFAR-100. This represents a significantly more challenging optimization landscape with a deeper architecture. We compared SSFL against the dense bascline (FedAvg) and the strongest sparse bascline (DisPFL) across the full sparsity spectrum (50% + 95%).
Table 1: Comprehensive comparison of SSFL with FL and sparse FL baselines on CIFAR-10 and CIFAR- 100 under a Dirichlet partition using ResNet18 at 50% sparsity. Highlighted values indicate the best performance.
Method
CIFAR-10 (%)
CIFAR-100 (%)
Comms (MB)
Sparse
FedAvg
86.04 1..3
59.30 +2.02
446.9
X
FedAvg-Fi FT
88.02 +0.57
59.42 +3.72
446.9
X
D-PSGD-FT
83.05 11.9
50.27 41.61
446.9
X
Ditto
83.50 +0.99
43.50 +2.04
446.9
X
FOMO
66.21 +1.54
32.50 43.65
446.9
X
SSFL
88.29 10.42
61.37 .2.01
223.4
DisPFL
85.12 1.05
59.21 11.97
224.0
SparsyFed
80.94 +1.20
50.64 +2.10
224.0
Flash
81.13 10.95
51.81 42.35
224.0
SubFed
76.50 +1.74
47.25 +2.84
346.6
Random
41.61 +1.62
48.62 +1.33
223.4
Fed-PM
46.44 +2250
15.88 +3.10
optimal
optimal
8

---

Published in Transactions on Machine Learning Research (12/2025)
Table Performance comparison on Pathological partition (C1FAR-10/100) and Dirichlet partition (Tinylm-
ageNet) using ResNet18 at 50% sparsity. Highlighted values indicate best performance.
Pathological Partition
TineImageNet (Dirichlet)
Method
CIFAR-10
CIFAR-100
Comms (MB)
Sparse
Method
Acc.
Comms (MB)
Sparse
(%)
(%)
(%)
FedAvg
92.48 10.25
52.47
12.68
446.9
FedAvg
16.92 10.43
446.9
X
FedArg-FT
92.90 +1.33
52.47
22.04
446.9
FedArg-FT
18.21 +0.48
446.9
X
D-PSGD-FT
88.74 +1.60
27.58
=1.88
446.9
X
D-PSGD-FT
11.98
+0662
446.9
X
Ditto
83.54 +1.28
51.90
=2.20
446.9
Ditto
17.80 +.339
446.9
X
FOMO
88.25 +1.10
45.25
12.09
446.9
FOMO
4.27
446.9
X
SSFL
94.61 11.00
52.01
11.4
223.4
SSFL
19.4 10.30
223.4
DisPFL
91.25 +1.10
44.74
11.5
224.0
DisPFL
8.27 +0.20
224.0
SubFedAvg
91.20 10.60
46.04
2289
346.6
SubFedAvg
18.76 10.28
346.6
Random
55.34 #1.25
46.22
+4.99
223.4
Random
16.76 +0.53
223.4
*Sinlle run; method failed converge.
The results in Table 3 demonstrate that SSFL's advantages amplity with the increase in model size when compared DisPFL, the best performing dynamic method in our analysis so far. SSFL consistently outperforms DisPFL across all sparsity levels, with the performance gap widening as networks become sparser from a +24.39% improvement at 50% sparsity to +33.49% at 95% sparsity. Notably, at 70% sparsity, SSFL achieves 60.82% accuracy, coming within 1.21% of the dense baseline while reducing communication by 3.3x. At extreme sparsity (95%), DisPFL's dynamic masking collapsed to 12.04%, while SSFL maintains good performance at 47.53%. These results indicate that stable, globally shared sparse subspace becomes increasingly critical as models scale and sparsity increases.
Table 3: ResNet-50 on CIFAR-100. SSFL significantly outperforms DisPFL, with the performance gap widening at higher sparsity levels.
Method
50%
70%
80%
90%
95%
FedAvg (Dense)
62.03%
62.03%
62.03%
62.03%
62.03%
DisPPL.
36.55%
32.96%
30.65%
21.87%
12.04%
SSFL (Ours)
59.76%
60.82%
56.73%
55.93%
47.53%
Improvement
+23.21%
+27.86%
+66.08%
+34.06%
+35.49%
Experiment Details and Basclines. We use the SGD optimizer for all techniques with a weight decay of 0.0005. All methods use 5 local epochs, except for Ditto, which uses 3 epochs for the local model and for the global model. The initial learning rate is 0.1, decaying by factor of 0.998 after each communication round. We use batch size of 16 for all experiments. A total of R 500 global communication rounds are executed for CIFAR-10 and CIFAR- C..R-100.FFO. Fo.Flooio.. standard practice (Dai al., 2022; Liu et al., 2025; Lin et al., 2018), we report communication cost under the values-only encoding assumption, where only non-zero weights are transmitted each round. This convention is conservative for SSFL, which broadcasts its global mask only once, while dynamic methods must repeatedly transmit mask indices, thereby incurring additional communication costs. We nevertheless adopt values-only encoding for all methods ensure fair and widely comparable benchmarks, with detailed overhead analysis provided Appendix A.4.
Our dense baselines include FedAvg (McMahan et al., 2017), FedAvg-FT (Cheng et al., 2021), Ditto (Li et al, 2021), FOMO(Zhang et al., 2020), and D-PSGD (Lian al., 2017). Our sparse baselines Dis-PFL(Dai et al., 2022), SubFedAvg (Vahidian et al., 2021), SparsyFed (Liu et al., 2025), Flash (Babakniya et al., 2023) and FedPM (Isik et al., 2022). Among the sparse baselines, FedPM (Isik et al., 2022) is unique in that it freezes the randomly initialized dense network and instead trains a probability mask over connections. The final subnetwork and sparsity emerge from this process rather than being presot, so we report its communication and sparsity as optimal in Table 1. A detailed explanation of the baselines and experimental setup available in Appendix A.S.
9

---

Published in Transactions on Machine Learning Research (12/2025)
Effect of varying levels of sparsity performance We evaluate how SSFL's performance varies across sparsity levels from 50% to 95% on CIFAR-10 and CIFAR-100, as shown in Figure 2. We compare SSFL against sparse FL baselines DisPFL, SubFedAvg, and FedPM, along with a random masking baseline. SSFL consistently outperforms all sparse baselincs across the entire sparsity spectrum. Notably, it maintains robust performance even at high sparsity levels where other methods degrade significantly, indicating that the global mask captures informative and transferable structure. Unlike DisPFL, which aggregates heterogencous masks into denser global model, SSFL's shared mask both simplifies training and preserves truly sparse global model that can further benefit from hardware acceleration (e.g., SPMM kernels on GPUs/ /TUUs and custom accelerators) (Nakahara et al., 2019; Thangarasa et al., 2023; Gale et al., 2020; NVIDIA, 2021; 2020a).
Performance at varying sparsity levels (CIFAR-10
Performance at varying Sparsity levels (CIFAR 100)
0.6
0.8
Accuracy
Accurace
0.4
SubFewtug
Random
Failay (Derse Baselinu)
FedPM
S5RL
DisFFL
0.2
Sandom
Rodgig (Densc Baseinel
RddP
0.0
50
50
80
90
50
90
65
70
80
90
Sparsity %
Sparsity %
(a)
(b)
Figure 2: Performance comparison at varying levels of sparsity for SSFL with similar sparse FL methods on R8 on the (a) CIFAR-10 and (b) CIFAR-100 non-IID dataset.
Wall-clock gains under real-world network conditions. We deployed experiments on Amazon Web Services (AWS) across five geographically distributed regions (North Virginia, Ohio, Oregon, London, and Frankfurt). This setup emulates realistic FL environment where clients experience substantial network latencies. We measured the average server aggregation time per round, i.e., the time required to collect all client model updates. As shown in Figure 3 (b), SSFL consistently reduces communication time relative to dense commuicattin (FedAvg), with the gap widening as model size increases. At 90% sparsity, SSFL achicves a speedup of approximately 2.3x on the 1.73M-parameter model, reducing the average aggregation latency from 1.91 to 0.82s per round. This highlights its practical impact in real-world deployments.
4.2 Structural Alignment and Robustness of the Global Sparse Mask
A central tenet of SSFL. is the enforcement of single, globally shared sparse structure discovered at initialization using limited data. We investigate the validity of this design choice by examining: ① the necessity of structural aligument compared to local masking @ the importance of specific mask topology versus layer-wise density 3 the sufficiency of our minle-mmnibatch estimation strategy and ! the mask's generalization across heterogeneous clients.
4.2.1 Global vs. Local Masks: Case for Structural Alignment
A natural alternative to SSFL is to allow each client to maintain its own sparse subnetwork. Indeed, using client-specific masks is intuitively appealing and has been popular choice in prior dynamic sparse training works (Dai et al., 2022; Liu et al., 2025; Evci et al., 2020a).
To isolate the importance of structural consensus, we conducted a controlled experiment comparing a shared global random mask against unique client-wise random masks (see Figure 3-a). The results demonstrate an interesting outcome: the global random mask significantly outperforms the client-specific local random masks. This highlights important insight: structural aligroment within a unified subspace is essendial for
10

---

Published in Transactions on Machine Learning Research (12/2025)
effective aggregation. By enforcing shared global mask, SSFL ensures that all clients operate within the same sparse subspace. This enables SGD to optimize consistent set of parameters from start to finish, avoiding the optimization instability and destructive interference caused by the haphazard aggregation of disjoint parameter updates from misaligned subspaces, which occurs in dynamic masking approaches.
4.2.2 Importance of mask topology: Effect of intra-layer mask permutation
While the previous experiment confirms the effoctiveness of a shared global structure, raises a sccondary question: does the specific lopology of the mask matter, or merely the sparsity ratios allocated to each layer? Previous work in centralized pruning (Frankle et al., 2021) suggests that for pruning-at at-inntialiiiiiionn identifying correct layer-wise densities is often the primary driver of performance. We test this in the non-IID FL setting by randomly shuffling the SSFL mask within each layer (preserving layer-wise density but destroying specific structural topology). As shown in Figure 3 (a), the shuffled variant outperforms the uniform global random mask, confirming that our aggregated saliency scores successfully identify optimal laser-wise capacitics. Importantly, however, the original, unshuffled SSFL mask consistently performs best, indicating that SSFL caplures meaningful information about individual connection importance beyond just. the layer-wise sparsity ratio.
4.2.3 Global Saliency Estimation via K Distributed Minibatches
SSFL requires only one local minibatch per client to construct the global sparse mask, but this does not mean the mask is computed from single minibatch overall. With K participating clients, the server aggregates saliency scores from K distinct minibatches, which in our cross-device setting (e.g.. K = 100) corresponds to several thousand samples, comparable to the data budgets used in singlo-shot pruning methods such as SNIP. Since saliency derived from first-order gradients, which concentrale rapidly with sample size, aggregating gradients across clients produces stable and low-variance estimate of the global saliency distribution, even under severe non-i.i.d. partitions.
Empirical validation. We further quantify this by approximating an oracle mask M computed using the full training set, and evaluating how well masks estimated from (1......00) minibatches recover M As shown in Figure 4 (a-b), the mask error decays sharply and plateaus after roughly S0 100 minibatches on both CIFAR-10 and CIFAR-100, with additional data yielding negligible improvements beyond this point. In our experiments, following the standard setting in the literature (Dai et al., 2022), we use K = 100 clients, meaning SSFI. aggregates exactly K = 100 minibatches (one per client) and thus operates prociscly in this convergence regime. This provides strong evidence that one minibatch per client is sufficient to recover a near-oracle sparse subnetwork at initialization when the client pool sufficienlyy large (K > 80), enabling
Bancom imye Stuffing
Communkation Time Compar ion CIFAR-10
2.25
(eedaag)
j0.rs
3.75
12-
30
2.2
0.75
nodien
Sponite leet
(a) SSFL vs. layer-wise shuffling
(b) Communcatiin time comparison
Figure Analysis of SSFL in different scenarios. (a) Effect of random intra-layer shuffling on SSFL masks. (b) Wall-time communication comparison between baseline dense communication and SSFL CIFAR-10 across ResNet models of varying complexities.
11

---

Published in Transactions on Machine Learning Research (12/2025)
0.18
3000
X
0.33
0.7
L500
008
00
Banches Processe2
Batches Processed
Charr.1
ClertiD
(a) CIFAR-10
(b) CIFAR-100
(c) Client accuracy
(d) Samples per client
Figure 4: (a-b) Mask Convergence. Mask error relative to the oracle decreases exponentially, stabilizing near K x 80-100 (vertical line). We use K 100 for all experiments. (c-d). Experiments done with random seeds. Client Performance. Local accuracy (c, bars) plotted against aggregation weights pk (orange line), alongside sample counts (d). SSFL maintains high performance even on minimal data partitions (e.g., Clients 8 and 9).
SSFL to achieve high accuracy without any iterative mask refinement or additional communication. The framework naturally extends to settings with fewer clients by increasing the number of minibatches per client. to maintain oracle mask approximation quality.
4.2.4 Quality of the discovered mask the non-IID setting.
We examine the effect of shared global mask on local model performance under uneven client data distributions. Figure 4 (c-d) shows that, despite large disparities in client data volumes, SSFL yields competitive accuracy even for clients with very limited data. This indicates that aggregated saliency scores capture structure that generalizes across heterogeneous clients rather than overfitting to high-data participants. We further confirm this Appendix A.10, which reports local accuracy alongside class distributions for random clients as well as the top and bottom-performing sites.
4.3
Out-of-Distribution (OOD) Adaptation
The one-time mask computation in SSFL facilitates adaptation to non-stationary environments, such as shifts in data distribution or the arrival of new clients with novel classes. To handle these scenarios, we extend SSFT, to One-Shot OOD Adaptation, a mechanism that triggers a mask refresh upon detecting distribution shifts. To identify the new optimal subnetwork, clients compute saliency scores the full local model state by temporarily unmasking pruned parameters, which retain their original initialization values. This approach allows the model to recruit connections outside the set of current mask if necessary for incoming data with negligible overhead compared to fully dynamic methods.
COD Disvo is Ouseet Adotarko Campe ton
Figure OOD classes are introduced round 225 (dotted vertical line). Fol- lowing the single mask update, the model rapidly acquires the new con- cepts (Red curve), rising from 0% to over 80% accuracy, while maintain- ing stable global performance (Green curve).
Figure validates this approach where two new OOD classes (classe 8 and 9) are introduced at round 225. Following the single mask update, the global model rapidly adapts to the new classes (Red curve), rising from 0% to over 80% accuracy, while maintaining stable global performance (Green curve). This alludes to SSFL accommodating to distribution shifts through a single, efficient topology update. Further analysis on OOD adaptation, including the OOD Algorithm is detailed in Appendix A.6 including direction of future work.
4.4 Further Analysis
We provide further analysis of SSFL's design choices, computational analysis, sensitivity warmup and practical deployment factors in Appendix A. Specifically, detail computational complexity in Appendix A.3
12

---

Published in Transactions on Machine Learning Research (12/2025)
and communication encoding costs in Appendix A.4, demonstrating SSFL's theoretical and empirical efficiency gains over dynamic methods. Additionally, Appendix A.7 investigates the sensitivity of saliency estimation to initialization by comparing our standard random initialization against warmup strategies.
5 Limitations and Future Work
While SSFL demonstrates the effectiveness of static sparse subnetwork discovery, several natural extensions remain. First, while we introduced a one-shot OOD adaptation mechanism to handle distinct distribution shifts, the core SSFL framework prioritizes stability via a static mask. In scenarios with highly volatile, continuous concept drift, more granular or automated adaptation strategics may be required balance the trade-off between mask stability and plasticity.
Second, our warmuu analysis (Appendix A.7) validates the robustness of initialization mask discovery, showing negligible performance differences (+1%) compared to post-warmup approaches. This finding reinforces the stability of our method, consistent with the low variance observed across experiments with five different random seeds. However, investigating alternative saliency criteria or ensembles beyond our gradient-basel approach may yield even stronger subnetworks in distributed non-lID settings.
Beyond these limitations, we identify several promising research directions. While our experiments on ResNet-50 demonstrate that SSFL's advantages over dynamic methods amplify with model scale (widening from +23% to +35% sparsity increases), extending this to foundation models remains exciting avenue. Furthermore, extending SSFL to structured sparsily patterns could unlock greater speedups on modern accelerators. SSFL naturally complements differential privacy, since it requires only a single noisy saliency aggregation step, thereby minimizing privacy breach while preserving communication efficiency. Finally, integrating personalization on top of a shared global mask may provide more robust performance under extreme heterogeneity across clients.
6
Conclusion
We presented Salient Sparse Federated Learning (SSFL), a streamlined framework that mitigates communica- tion bottlenecks federated learning by identifying high-quality global sparse subnetwork before training. SSFL requires only single, privacy-preserving round of communication where local gradient-based saliency scores are aggregated into shared mask. This static initialization eliminates the hyperparameter tuning and iterative coordination typical of dynamic sparsity methods.
SSFL achieves substantial communicttion savings across CIFAR-10, CIFAR-100, and matching or exceeding dense baselines. More importantly, we demonstrated that these benefits scale to deeper architectures (ResNet-50), where SSFL maintains superior accuracy-sparsiy, trade-offs compared to dynamic approache DisPFL, Furthermore, our introduction of a one-shot OOD adaptation mechanism shows that SSFL has the flexibility to readily handle distribution shifts without abandoning the efficiency of static mask during training.
while
Real-world deployment on geographically distributed AWS regions further shows to 2.3x wall-clock speedups, underscoring practical efficiency. These results challenge the assumption that complex, itcrative mask adjustments are required for strong sparse FL performance. Instead, they demonstrate that a well-chosen static subnetwork, identified collectively at initialization, enables a simpler, more stable, and highly effective training trajectory. Looking ahead, SSFL offers robust foundation for hybrid static adaptive refinements, personalized sparse models, and structured sparsity for hardware acceleration.
13

---

Published in Transactions on Machine Learning Research (12/2025)
7
Broader Impact
This paper presents work whose goal is to advance the field of Machine Learning. Our primary objective is to study and improve efficiency associated with distributed federated learning.
Broader Impact Statement
This work aims to improve the efficiency of federated learning by reducing commuication and computation costs. More efficient FL can broaden access to collaborative training in settings with limited resources, such as mobile or hearthcare applications, and may help reduce the environmenial foolprint of large-scale distributed training. As with any federated approach, care required to ensure privacy protections (e.g., differential privacy or secure aggregation) and to prevent efficiency gains from coming at the expense of marginalized users.
Acknowledgments
We thank the action editor and the reviewers for their insightful comments that helped us improve the paper. This work was supported by NIH R01DA040487 and in part by NSF 2112455, and NIH 2R01EB006841
References
Samin Yeasar Arnob, Riyasat Ohib, Sergey Plis, and Doina Precup. Single-shot pruning for offline reinforcement learning. arXiv preprint arXim:212. 15579, 2021.
Samin Yeasar Arnob, Riyasat Ohib, Sergey Plis, Amy Zhang, Alessandro Sordoni, and Doina Precup. Efficient reinforcement learning by discovering neural pathways. Advances in Neural Information Processing Systems, 37:18660 18694, 2024.
Samin Yeasar Arnob, Zhan Su, Minseon Kim, Oleksiy Ostapenko, Riyasat Ohib, Esra'a Saleh, Doina Precup, Lucas Caccia, and Alessandro Sordoni. Exploring sparse adapters for scalable merging parameter efficient experts. In Second Conference on Language Modeling, 2025. URL https: http://penrreieeeeee/eoeooo te7UC872bw.
Sara Babakniya, Souvik Kundu, Saurav Prakash, Yue Niu, and Salman Avestimehr. Federated sparse training: Lottery aware model compression for resource constrained edge. arXie preprint arXiu:228.. 13092, 2022.
Sara Babakniya, Souvik Kundu, Saurav Prakash, Yue Niu, and Salman Avestimehr. Revisiting sparsily hunting in federated learning: Why does sparsity consensus matter? Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL httpp://pppparrreeeaeeesettttttp
Leighton Pate Barnes, Huseyin Inan, Berivan Isik, and Ayfer Ozgür. rtop-k: statistical estimation approach distributed sgd. JEEE Journal on Selected Areas in Information Theory, 1(3):897 907, 2020.
Mahdi Beitollahi, Mingrui Liu, and Ning Lu. Dafl: Dynamic sparsification for federated learning. 2022 5th International Conference on Communications, Signal Processing, and their Applications (ICC- SPA), pp. 1 6. IEEE, 2022. doi: 10.100//CCSPAA5560.2222 10019204. URL httpp:://bo.org//00119 ICCSPA55860 202.
Jeremy Bernstein, Jiawei Zhao, Kamyar Azizzadenesheli, and Anima Anandkumar, signSGD: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pp. 560 569, 2018.
Sameer Bibikar, Haris Vikalo, Zhangyang Wang, and Xiaohan Chen. Federated dynamic sparse training: Computing less, communicating less, yet learning better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 1080-6088) 2022.
Cerebras. Wafer Scale Engine: Why We Need Big Chips for Deep Learning, https:///orebbra.nnt/bbbb Ccrebras-vafen scwle-alleeeeeeee wwwweennnn----eeeeeee neeee--eee-eeee meed-big- senddbig-chhieee do-p----oo--ee--eeerne 2019.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33:15834 15846, 2020.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16306-16316, 2021.
Gary Cheng, Karan Chadha, and John Duchi. Fine-tuning fine in federated learning. arXiv preprint arXiu:2108.07913, 2021.
Francesco Curci, Luca Griggio, Arco Aarts, and Decebal Constantin Mocanu. Nerva: A truly sparse implementation of neural networks. arXiv preprint (tt::4077174377 2024.
Rong Dai, Li Shen, Fengxiang He, Xinmei Tian, and Dacheng Tao. Dispfl: Towards communication-efliieet personalized federated learning via decentralized sparse training, arXiv preprint arXiv:2206. 0tl::206000187,, 2022.
Pau De Jorge, Amartya Sanyal, Harkirat S Behl, Philip HS Torr, Gregory Rogez, and Puneet K Doka- nia. Progressive skeletonization: Trimming more fat from a network at initialization. arXiv preprint wrXi:2000.09055) 2020.
Enmao Diao, Jie Ding, and Vahid Tarokh. HeteroFL: Computation and communication efficient federated learning for heterogeneous clients. In International Conference on Learning Represenlations, 2021.
Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. Fast sparse convnets. In Proceedings of the IREE/CVF conference on computer vision and pattern recognition, pp. 14629-14633, 14638, 2020.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943 2252. PMLR, 2020a.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning (ICML), pp. 2943-29222 2952. PMLR, 2020b. URL https:// httpas///roceeeing..l......
Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. Fate-llm: A industrial grade federated learning framework for large language models. arXiv preprint ar arXin:2310.1004 2023.
Wenshuo Fan, Kaiping Yang, Yongxin Wang, et Data-frce adaptive structured pruning for foderated learning. The Journal of Supereomputing, 80:18600 18626, 2024.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: with simple and efficient sparsity. Journal of Machine Learning Research, 23(12)): 1-3,, 2022.
Scaling to trillion parameter models
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint XrX:: 01611, 2019. This work was later subsumed by 'Linear Mode Connectivity and the Lottery Ticket Hypothesis'.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259-3269. PMLR, 2020a.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis, 2020b. URL https: http://aaivv.../ga/////00.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? In 9th International Conference on Learning Represenlations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Karel Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In International Conference Machine Learning (ICML), 2023. URL http:///aaaa....rraa///.....1..
Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. CoRR, abs/2006.10901, 2020. URL
Graphcore.
Training sparse, large-scale language models on Graphcore's IPU. 2023.
Adriano Guastella, Lorenzo Sani, Alex Iacob, Alessio Mora, Paolo Bellavista, and Nicholas D. Lane. SparsyFed: Sparse adaptive federated training. In International Conference on Learning Representations (ICLR), 2025.
URL /tap:::///rxxvv www.//oggg//////eeeeeee 0abbs//554005555
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXi::151000014, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1026-1034, 2015.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Mcasuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv: 1rXii:1909.0000 06390 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
Hong Huang, Lan Zhang, Chaoyue Sun, Ruogu Fang, Xiaoyong Yuan, and Dapeng Wu. Fedtiny: Pruned federated learning towards specialized tiny models. arXiv preprint wxXi::222.00197 2022.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accelerated sparse noural training: A provable and efficient method to find N:M transposable masks. Advances in Newral Information Processing Systems, 34:21099 21111, 2021.
Berivan Isik, Francesco Pase, Deniz Gunduz, Tsachy Weissman, and Michele Zorzi Sparse random networks for commmication-effiieen federated learning. arXiv preprint rxXiw:2209.15322 2022.
Steven A Janowsky. Pruning versus clipping in neural networks. Physical Review A, 39(12):6600, 1989.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint Xiv::24411 Xix:2400.04088, 2024.
Yuang Jiang, Shiqiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K Leung, and Leandros Tassiulas. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Peler Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in foderated learning. Foundations and TrendsS in Machine Learning,
2021.
Mikhail Khodak, Renbo Tu, Tian Li, Liam Li, Maria-Florina F Balcan, Virginia Smith, and Ameet Talwalkar. Federated hyperparameter tuning: Challenges, baselines, and connections to weight-sharing, Advances in Neural Information Processing Systems, 34:19184-19197 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. University of Toronto, 2009.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Eldar Kurtić, Elias Frantar, and Dan Alistarh. Ziphm: Inference-aware structured pruning of language models. Advances in Newral Information Processing Systems, 36:65597-656) 2023.
Mustafa Lasby, Sharath Nittur Kumar, Ilias Leontiadis, Luca Griggio, Raghuraman Krishnamoorthi, Ritchie Zhao, and Vikas Chandra. Structured pruning learns compact and accurate models. In Advonces in Neural Information Processing Systems, volume 36, 2023.
Mustafa Lasby, Yunyang Shen, Raghuraman Krishnamoorthi, Ritchic Zhao, Krishnateja Killamsstty, Carole- Jean Wu, Alexander Wettig, Vikas Chandra, Mehrdad Khani, Moses Kumbong, and Kasra Khosravi. Dynamic sparse training of diagonally sparse networks. arXiv preprint arXiv:2506.11449 2025.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-6008-00 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint orXiv:1810.02344 2018.
Ang Li, Jingwei Sun, Binghui Wang, Lin Duan, Sicheng Li, Yiran Chen, and Hai Li. Lotteryfl: Personalized federated learning with lottery ticket hypothesis on non-iid datasets. arXív
and commuication--ffiee preprint Xiv: 03371, 2020.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608. 087100, 2016.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357-6368. PMLR, 2021.
Zhengming Li, Jiahui Chen, Peifeng Zhang, Huiwu Huang, and Guanbin Li. Dafedcon: Dynamic sparse federated contrastive learning for data-driven intelligent systems. IEEE Transactions on Neural Networks and Learning Systems, 36(2):3343 (32):3333--335, February 2025. doi: 00.100//NNNIS URL https:
Zhiyi Li, Douglas Orr, Valeriu Ohan, Tom Murray, Adam Sanders, Deniz Beker, Dominic Masters, al. Popsparse: Accelerated block sparse matrix multiplication on ipu. arXio preprint urXio:2303116993 2023.
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsich, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in Neural Information Processing Systems, 2017.
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv preprint arXiv:1712.01887, 2017.
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. In International Conference on Learning Representations, 2018.
Yang Liu, Yi Zhao, Guangmeng Zhou, and Ke Xu. Fedprune: personalized and communicationn-fficcentt federated learning on non-iid data. In Neural Information Processing: 28th International Conference, ICONIP 2021, Sanar, Bali, Indonesio, December 8 8-12, 2021, Proceedings, Part V 28, pp. 2021.
Springer,
Yue Liu, Tao Chen, and Li Wang. SparsyFed: A pearitty-aware federated learning framework with periodic rewiring. In To appear in Proceedings of the 19th International Conference on Learning Representations, ICLR 2025, 2025.
Rongrong Ma, Janyy Miao, Lingfeng Niu, and Peng Zhang. Transformed & regularization for learning sparse deep neural networks. Neural Networks, 119:286 298,6299881 2019.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Lim-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 06:21702-22 21720, 2023.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Brendan McMahan and Daniel Ramage. Federated learning: out centralized training data, 2017. URL http:://....oooog.o..c/////////a collaborative.ttm.
Collaborative machine learning with-
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communictiion efficient. learning of deep networks from decentralized data. In Arlificial intelligence and slatistics, pp- PMLR, 2017.
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatosh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. Xr preprint urXiv:210,.08376, 2021.
Ari S Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: Generalizing lottery ticket initializations across datasets and optimizers. In Advances in Neural Information Processing Systems, pp. 1932-4942, 2019.
Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a net via relevance assessment. Advances in neural information processing systems, 1, 1988.
Muhammad Tahir Munir, Muhammad Mustansar Saeed, Mahad Ali, Zafar Ayyub Qazi, and Ihsan Ayyub Qasi. Fedprune: Towards inclusive lederated learning. orXiv preprinl ar 2tt:2110.1444 2021.
H. Nakahara et al. Fpga-based training accelerator utilizing sparseness of convolutional neural network. Tn 2019 29th International Conference on Field Programmable Logic and Applications (FPL), pp. 180 186, 2019. doi:
NVIDIA. How sparsity adds umph to ai inference. http:////haaaaaaaaaoo/////////// ai-inference/, 2020a.
NVIDIA. Accelcrating sparse deop neural networks.
NVIDIA Technical Blog, 2020b. URL
https://ddeveloper.nvv htt.:/ppr..viiia.coom//bb http:://hh..ccceeeeeeeeoetteeeee ampere ampere- and-tensorrt/.
NVIDIA. Accelerating matrix multiplication with block sparse format and nvidia tensor cores. http://deeeelppe...iiiaa..a..aaa awwaccceeeainnnn mat.iiimmiiiiiiiiiiioooooooo a wtt.diiiiaattssooooreeeeeee 2021.
NVIDIA Corporation.
Exploiting NVIDIA Ampere structured sparsity with cuSPARSEIt.
2022.
NVIDIA Corporation. Accelerating matrix multiplication with block sparse format and NVIDIA tensor cores.
http:////www.....oooo//////tttt/ ividia-tensor-cores// 2023a.
NVIDIA Corporation. Accelerating inference with sparsity using the NVIDIA Ampere architecture and NVIDIA TeTsorRT.. htlps://developer. wttp://elllee.niiia....omm and-tensorrt/, 2023b.
Riyasat Ohib. Explicit group sparse projection for machine lcarning. Master's thesis, Georgia Institute of Technology, 2023.
Riyasat Ohib, Nicolas Gillis, Sameena Shah, Vamsi Potluru, and Sergey Plis. Grouped sparse projection for deep learning.
Riyasat Ohib, Nicolas Gillis, Niccolo Dalmasso, Samoena Shah, Vamsi K. Potluru, and Sergey Plis. Explicit group sparse projection with applications deep learning and NMF. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL http:://ppearrreeeeeeee/c/rrdddp
18

---

Published in Transactions on Machine Learning Research (12/2025)
Riyasat Ohib, Bishal Thapaliya, Pratyush Gaggenapalli, Jingyu Liu, Vince Calhoun, and Sergey Plis. Salientgrads: Sparse models for communication efficient and data aware distributed federated training. arXiv preprint orXic:2304.07488, 2023.
Zeyneddin Oz, Ceylan Soygul Oz, Abdollah Malekjafarian, Nima Afraz, and Fatemeh Golpayegani. submfl: Compatiple submodel generation for federated learning in device heterogenous environment, 2024. URL
Jeff Pool. Accelerating sparsity in the NVIDIA Ampere architecture. GTC 2020 Presentation, 2021. Available at https://deeelpprr..ommbaaaaddddd cot/iiieoooo/o/////oo pttpecccooofftt/////////aaaaaaaa htt.:///ggg.ggo//////////////aan spaeriiyiirrhleeemmddddiieeeeeee
Jeff Pool, Abhishek Sawarkar, and Jay Rodge, Accelerating Inference with Sparsity Using the NVIDIA Ampere
Architecture and NVIDIA TensorRT. https://develop: www.da..coobbaaaeeaa www.daaa.coom//lgggaaaaaaataaaaa inference-
wi
aagpgaggaampr------ttessorrr 2021.
PyTorch Team. Accelerating noural network training with semi-structured (2:4) sparsity.
huups//pptoorch.or//bbo//// http:///wwwwoooooo/o/////////eca
2024.
Russell Roed. Pruning algorithms-a survey. IEEE transactions on Neural Networks, 4(5):740-747,, 1993.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint arXi::2003. 023899 2020.
Sejin Seo, Seung-Woo Woo Ko, Jihong Park, Seong-Lyun Kim, and Mehdi Bennis. Communcattion--ffcieeett and personalized federated lottery ticket learning. In 2021 IEEE 22nd International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), pp. 581-585. IEFF, 2021.
Mehdi Setayesh, Xiaoxiao Li, and Vincent WS Wong. Perfedmask: Personalized federated learning with optimized masking vectors. In The Fleventh International Conference on Learning Representations, 2022.
Ghada Sokar, Elena Mocanu, Decebal Constantin Mocamu, Mykola Pechenizkiy, and Peter Stone. Dynamic sparse training for deep reinforcement learning. arXiv preprint orXi::2106.042. 2021.
Shubham Soni, Mike Davies, Saransh Yanamandra, Swagath Venkataramani, and Anand Raghunathan. Region masking to accelerate vidco processing on ncuromorphic hardware. orXie preprint arXiv:2503.10% 16775, 2025.
Mingjic Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprinl Xiv:22006. 1 11695, 2023.
Hidenori Tanaka, Daniel Kunin, Daniel L. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Floring Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annuual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
Vithursan Thangarasa, Abhay Gupta, William Marshall, Tianda Li, Kevin Leong, Dennis DeCoste, Sean Lie, and Shreyas Saxena. Spdf: Sparse pre-training and dense fine-tuning for large language models. arXiv preprint Xrxi:22303. 10464, 2023. URL
abs/ //////33..144 Accepted to UAI 2023.
Bishal Thapaliya, Riyasat Ohib, Eloy Geenjaar, Jingyu Liu, Vince Calhoun, and Sergey M Plis. Efficient federated learning for distributed neuroimaging data. Frontiers in Neuroinformatics, 18:1430987, 2024.
Saeed Vahidian, Mahdi Morafah, and Bill Lin. Personalized federated learning by structured and unstructured pruning under data heterogeneity. arXiv preprint arXiv:2105. 00562, 2021.
Chaoqi Wang, Guodong Zhang, and Roger B. Grosse. Picking winning tickets before training by preserving gradient flow. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababo, Ethiopia, April 26-30, 2020, 2020.
19

---

Published in Transactions on Machine Learning Research (12/2025)
Haonan Wang, Zeli Liu, Kajimusugura Hoshino, Tuo Zhang, John Paul Walters, and Stephen Crago. FedPal: Achieving extreme sparsity in federated learning via pruning at initialization. or preprinl arXiu:2504.00308 2025.
Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Terngrad: Ternary gradients to reduce communication in distributed deep learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
Yuxin Wu and Kaiming Hc. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp- 3 3-19, 2018.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pp. 38087-38099. PMLR, 2023.
Jintao Yan, Tan Chen, Yuxuan Sun, Zhaojun Nan, Sheng Zhou, and Zhisheng Niu. Nobility-aware asyn- chronous federated learning with dynamic sparsification. arXiv preprint Mixx:22506.03222 2025. URL
Huanrui Yang, Wei Wen, and Hai Li. Deephoyer: Learning sparser neural network with differentiable scale-invariant sparsity measures. arXiv preprint 19X8: 09999, 2019.
Sixing Yu, Phuong Nguyen, Ali Anwar, and Ali Jannesari. Adaptive dynamic pruning for non-iid federated learning. arXiv preprint arXiv:2 20002000.691, 2021.
Sixing Yu, Pablo Muñoz, and Ali Jannesari. Bridging the gap between foundation models and heterogeneous federated learning. aiti preprint arXiv:2310.003: 2023.
Michael Zhang, Karan Sapra, Sanja. Fidler, Serena Yeung, and Jose M Alvarez. Personalized federated learning with first order model optimization. arXiv preprint XXXi:20122085633 2020.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakii, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-cffciient fine-tuning. arXiv preprint 200: 10512, 2023.
Guoheng Zhou, Kun Xu, Qingyuan Li, Yang Liu, and Yi Zhao. Adaptel: Efficient collaborative learning with dynamic and adaptive pruning. arXiv preprint arXiv:2106. 14126, 2021.
Michael Zhu and Suyog Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In 6th International Conference on Learning Representations, ICLR 2018, Vancowver, BC, Canada, April 30 May 3, 2018, Workshop Track Proceedings, 2018.
20

---

Published in Transactions on Machine Learning Research (12/2025)
A Appendix
A.1 Detailed analysis of the Connection Saliency Criterion
This section provides a detailed derivation of the gradient-based connection saliency metric used in SSFL, as defined in Equation (1). The fundamental goal of pruning at initialization is to identify the most important parameters in a randomly initialized network, wo, without performing any training. This line of work traces back to the carly idea of skcletonization by Mozer and Smolensky (Mozer & Smolensky, 1988) and was later refined in single-shot pruning methods such SNIP (Lee al., 2018).
The importance of single parameter, wo,j. be defined the change in the total loss, C, that would occur that parameter were removed from the network. Formally, this change ALz is:
AC; = L(ws; D) - [(wo ey,
(4)
where ej is the one-hot vector for the j-th parameter. However, computing this value for every parameter is compulationally prohibitive, it would require d d forward passes over the data D, where d is the number of parameters.
To create tractable importance score, we approximate ACj using first-order Taylor expansion. This approach reframes the question from "what is the effect of removing a parameter?" to "how sensitive is the loss to the presence of a parameter at initialization?". To formalize this, we introduce an auxiliary 'gating' vector c E [0, 1]d, which multiplies the model's weights wo. The loss is now a function the masked weights: C(c @ wo: D).
The sensitivity of the loss to the j-th paramcter be measured by the derivative of L with respect to Cj, evaluated at = (the initial state where all connections are fully active). This derivative, which we denote as the saliency 9y: approximates the effect of small perturbation on connection j. We can compute it directly using the chain rule:
aC(c 3 wo: D)
9, (, D)
(5)
Dej c=
aL(wo; D)
(cywoo.))
(6)
iwo.j
Ocy
c=1
aL(wo; D)) Wo.i
(7)
iwo.i
This result, 9j. provides computationally efficient, single-pass estimate of the importance of parameter Wo The final saliency score 8j used in SSFL is the magnit ude of this term, preserving parameters that have the largest estimated impact on the loss:
aL D) Dwo.i
8, 1g, (wo; D)
woos
(8)
It is important to note that while these scores could be normalized (e.g., by dividing by their sum), this step is unnecessary for our method, the Top-k selection procedure only depends the relative ranking of the scores.
A.2 Generating non-IID data partition with Dirichlet Distribution
In this section, we provide the necessary background generating non-identical data distribution in the client sites using the Dirichlet Distribution, specifically for the context of federated learning.
non-IID data in FL Federated Learning (FL), as introduced by McMahan & Ramage (2017), is a framework designed for training models on decentralized data while preserving privacy. It utilizes the Federated Averaging (FedAvg) algorithm where each device, or client, receives model from a central server, performs stochastic gradient descent (SGD) its local data, and sends the models back for aggregation.
21

---

Published in Transactions on Machine Learning Research (12/2025)
Unlike data-center training where data batches often IID (independent and identically distributed), FL typically deals with non-IID) data distributions across different clients. Hence, to evaluate federated learning it is crucial to not make the IID assumption and instead generate non-llD data among clients for evaluation Hsu et al. (2019).
Generating non-IID data from Dirichlet Distribution In this study, we assume that each client independently chooses training samples. These samples are classified into N distinct classes, with the distribution of class labels governed by a probability vector q, which is non-negative and whose components sum to that is, q > 0, i E [V and llq 1. For generating a group of non-identical clients, q Dir(ap) is drawn from the Dirichlet Distribution, with p charactcrizing a prior distribution over the N classes and a controls the degree of identicality among the existing clients and is known as the concentration parameler.
this section, we generate a range client data partitions from the Dirichlet distribution with a range of values for the concentration parameter for exposition. In Figure 6, we generate a group of 10 balanced clients, each holding equal number of total samples. Similar to Hsu al. (2019) the prior distribution p is assumed to be uniform across all classes. For each client, given concentration parameter a, we sample q from Dir(a) and allocate the corresponding fraction of samples from each client to that client. Figure 6 illustrates the effect of the concentration parameter distribution on different clients, for the CIFAR-10 dataset. When a + 0o, identical class distribution is assigned to each classes. With decreasing a, more non-identicalness is introducod the class distribution among the client population. At the other extreme with a - 0, each class only consists of one particular class.
the class distribution drawn from the Dirichlet
Non-IID Client Data Partitions using Dirichlet Distribution for CIFAR-10 [10 clients, 10 classes)
Diriowet, di+
Dirkclet, a- 100.0
Dirichlt,, e- 1.0
Dirchlet,
DiKhlet,
010
C30
C10
C30
C10
08-
C8-1
C8
06
05-
05-
C3
C3
CL-
C2
0.
32
0.4
0.5
0.8
1.0
04
02
0.4
0.8
1.
0.0
0.2
D-4
0.8
0.5
34
0.
0.2
64
0.6
0.8
0.
0.2
0.5
0.5
1.0
Class buttribttion
Class distribution
ass distribution
Class distribution
Cless distribution
Class
Class Clvos
Closs 4
Cless 6
Clsss Cane
Clave
Clovk 1
Closs >
Figure 6: Generating non-identical client data partitions using the Dirichlet Distribution for the CIFAR-10 dataset among 10 clients. Distribution among classes is represented using different colors. (a) Dirichlet, a + oc results in identical clients (b-d) Client distributions generated from Dirichlet distributions with different concentration parameters (e) Dirichlet, - 0.0 results in each client being assigned only one class.
A.3 Computational Complexity and Communication Analysis
To rigorously evaluate the efficiency of SSFL, we analyze the computational cost under two distinct execution regimes: Standard Dense Execution and Accelerated Sparse Execution. Let P denote the number of model parameters, E the number of local epochs, B the number of batches per epoch, and E (0, 1] the density of the model. We decompose the total cost per training round into three components: gradient computation (Cgraa), dynamic topology overhead (Coverhead)) and communication cost.
Dynamic sparse training methods ineur structural overheads to update masks, such as sorting, pruning, and regrowth that often bottleneck accelerated execution. SparsyFed incurs high overhead, as it applies
22

---

Published in Transactions on Machine Learning Research (12/2025)
weight re-parameterriattin and activation pruning during every forward pass of every batch, scaling Coverhead R O P). Iterative pruning methods like DisPFL. and the FLASH-JMWST variant rely on periodic prune-and-regrow cycles that require magnitude- based sorting of weights, resulting in a moderate overhead of Coverhead = O(Plog P) per round. In contrast, SSFL (Ours) and FLASH-SPDST utilize static mask. Consequently, the lopology remains fixed during the training phase, eliminating dynamic overhead entirely (Coverhead = 0).
A critical distinction lies in the initialization cost. While FLASH-SPDST shares the per-round efficiency of static mask, it requires computationally expensive " "Warm-up" Stage involving Eg epochs of training subset of clients to estimate scnsitivity. Conversely, SSFL is single-shot, estimating saliency using only one minibatch per client at initialization (requiring K ~ 80 - 100 clients for mask convergence, a constant factor). Since 1 < Ed, SSFL significantly more lightweight to deploy.
In terms of communication bandwidth, SSFL transmits only the non-zero values (sP) in the uplink since the indices are fixed and globally synchronized. Dynamic methods such as DisPFL, FLASH-JMWST, and SparsyFed lack this global index synchronization, requiring the transmission of both values and updated mask indices (sP Idx). Uniquely, because SSFL and FLASH-SPDST enforce a fixed global mask, the server broadcasts only the updated sparse values (sP), achieving symmetric sparsity in the downlink. In contrast, DisPFL, and FLASH-JMWS, typically require synchronizing dense or dynamic structures, increasing the downlink load to O(P) or requiring index transmission.
Table 4: Per-round computational complexity and communication costs. SSFL achieves the minimum accelerated compute cost and communication volume without the dynamic overhcads of SparsyFed/IMWST or the initialization warm-up cost of FLASH-SPDST.
Accelerated Compute
Dynamic
Init.
Communication
Method
Overhead
Cost
Uplink
Downlink
FedAvg
O(1-P)
None
O(P)
O(P)
SSFL
O(s- P)
Low
O(sP)
O(sP)
SparsyFed
O(s-P)
O P)
None
O(sP) Idx
O(sP) Idx
DisPFL
P))-P)
O P)
None
O(sP) + Idx
O(sP) + Idx
FLASH-JMWST
O(s-P)
O(P log P)
High
O(sP) + Idx
O(sP) +Idx
FLASH-SPDST
O(s- P)
High
O(sP)
O(sP)
"FLASH-SPDST//MMST require 'Warm-up" Stage (Ea epochs training), whereas SSFL is single-shot or constant minibatch.
A.4 Communication Encoding Overheads
In the main body, we adopt the standard values-only encoding assumption (counting only non-zero weight values, not their structural indices), consistent with prior work on sparsity and federated optimization (Dai et al., 2022; Liu et al., 2025; Lin et al, 2018). This isolates the savings due to sparsity itself, while recognizing that efficient communication and encoding schemes are orthogonal line of work (Wen et al., 2017; Bernstein et al., 2018). Here, we quantify the specific overheads of dynamic topology transmission.
Let P be the number of model parameters and 8 E (0, be the density (fraction of non-zero elements). A dense model transmitted in fp32 requires: Cost = 4P bytes.
For sparse models, dynamic methods must transmit the topology. Under coordinate (COO) encoding, the cost for k sP non-zeros is:
Costcoo 4k (vals) + 4k (idxs) 8sP bytes.
23

---

Published in Transactions on Machine Learning Research (12/2025)
Bitmask encoding sends dense binary mask of length P alongside the values:
P Costbitmask 4k 4 4sP 0.125P bytes.
Dynamic sparsity methods (e.g., DisPFL, SparsyFed) must pay one of these overheads at every round to update the mask. In contrast, SSFL discovers global mask once at initialization. After this one-time broadcast, the lopology is static, and clients strictly transmit values:
COsIsSFI. 4k: = 4&P
byles.
Table compares these costs. At moderate density (s 0.5), SSFL matches the theoretical limit of sparse compression. Crucially, at high sparsity (s 0.05), SSFL achieves the theoretical minimum cost (5.0%), outperforming bitmasks (8.1%) which are limited by fixed overhcads, and maintaining a 2x advantage over COO encoding (10.0%).
Table 5: Communication cost (bytes) under different encoding schemes. SSFL achieves the minimal theoretical cost (4sP) by climinating the recurring structural overhead required by dynamic methods.
Encoding scheme
Formula
Cost at = 0.5 Cost at 0.05
Dense
4P
100.0%
100.0%
Bitmask (Dynamic)
AsP + P/8
53.1%
8.1%
COO (Dynamic)
8sP
100.0%
10.0%
SSFL (Fixed Mask)
4sP
50.0%
5.0%
A.5 Static Sparsity and Hardware Acceleration
A critical advantage of SSFL's static masking approach is its compatibility with hardware acceleration on modern AI accelerators. Static sparsity patterns enable substantial performance gains that are difficult or impossible to achieve with dynamic sparse training methods.
Hardware Support for Static Sparsity. Modern Al accelerators increasingly provide native support for sparse operations with fixed patterns. NVIDIA's Ampere and Hopper architectures introduced Sparse Tensor Cores (Mishra et al., 2021) that exploit 2:4 structured sparsity to achieve 2x math throughput compared to dense operations. Similarly, Graphcore's Intelligence Processing Units (IPUs) provide optimized kernels for static block sparsity (Li et al., 2023), demonstrating that static sparse implementations can outperform dense equivalents at 90% sparsity. These hardware optimizations rely fundamentally on knowing the sparsity pattern compile time, enabling efficient memory layout, vectorized operations, and optimized instruction scheduling (NVIDIA, 2020b).
Why Static Masks Enable Acceleration. Static sparsity patterns provide three key advantages for hardware acceleration: (1) Compile- time optimization: fixed masks allow compilers to generate optimized code with predictable memory access patterns and efficient sparse data structures (Gale et al., 2020); (2) Zero runtime overhead: the mask need not be recomputed, compressod, or communicated during training, eliminating dynamic topology management. costs (PyTorch Team, 2024); and (3) Hardware-frieudly formats: static patterns can be stored in compressed formats (e.g., CSR, blocked layouts) that align with specialized hardware instructions (NVIDIA Corporation, 2023b;a). For instance, NVIDIA's cuSPARSELt library achieves near-linear speedup with sparsity when using static patterns, as the compressed representation can be pre-computed and the metadata efficiently organized for Tensor Core operations (NVIDIA Corporation, 2022).
Limitations of Dynamic Sparsity for Hardware Acceleration. In contrast, dynamic sparse training methods face fundamental barricrs to hardware acceleration. Dynamic methods must repeatedly prunc and regrow connections during training, which introduces several sources of overhead: recomputation: dynamic methods incur O P) sorting costs O(B P) per-batch costs for topology
(1) Runtime mask
24

---

Published in Transactions on Machine Learning Research (12/2025)
updates Evci et al. (2020b); Guastella et al. (2025), where P is the parameter count and B is the number of batches; (2) Control-flow bottlenecks: changing sparse patterns break vectorization and prevent efficient use of specialized instructions Hubara et al. (2021); Lasby et al. (2023); (3) Memory access irregularity: dynamic topology changes lead to umpredictable, scattered memory accesses that underutilize memory bandwidth Gale et al. (2020); and (4) Metadata overhead: dynamic methods must repealedly compress, decompress, and communicate mask indices, often requiring metadata storage that exceods the weight storage itself Pool (2021).
Empirical evidence confirms these limitations. Hardware studies show that dynamic sparse patterns fail to achieve practical speedups on GPUs and specialized accolerators Lasby et al. (2025); Curci et al. (2024). Neuromorphic computing research demonstrates that static masks achieve up to 2.06x better energy- delay product than dynamic masks specifically because they eliminate additional computation and data movement Soni et al. (2025). Even when dynamic methods achieve theoretical FT.OP reductions, these do not translate wall-clock speodups due to the overhead of mask management Graphcore (2023).
Compatibility with Accelcrators. By discovering a global mask once and maintaining it throughout training, SSFL's design allows it to be compatible with the same hardware optimizations available for sparse inference, during the local training epochs. The fixed mask can be compressed offline, stored in hardware-optimiized formats, and exploited by specialized kernels GPUs (via DOSPARRSE/uSPARRSIIE TPUs, and custom accelerators Nakahara et al. (2019); Thangarasa et al. (2023). This positions SSFL not only as communication-effiienn but also as a pathway to training-time acceleration on next-generation sparse-optimized hardware, an advantage that dynamic methods cannot easily match.
A.6 Adaptation to Out-of- DfDtsriiibutiin (OOD) Shifts
Resilience non-stationary environments, such as the arrival of clients with novel data classes, is critical for robust. federated learning. While SSFL. focuses on stable global sparsily for efficiency, the framework allows for adaptation through OOD Adaptation when new clients with different data distribution joins the fodcrated learning scheme. This approach triggers mask rediscovery when distribution shifts are detected, avoiding the instability and overhead continuous dynamic updates.
Algorithm 2 SSFT, Extension: OOD Adapiation
Require: Clients Ktotal, current model wy, current mask m, sparsity level Ensure: Updated mask maew and adapted model wr 1: // Note: Wr contains trained values for active params and init values for pruned params
// Phase 1: Single-Shot Saliency Re-aggregation
for each client k Krotal in parallel do Sample minibatch B₁ D
6: Compute local saliency using Equation (1) on wr
7: Send 8y and n4 |Dx] to server
8: end for
9:
10: // Phase Server-Side Mask Update
11: Server computes data proportions: Pk = n4/ ΣK
1== n4
12: Server aggregates global saliency: Sglobal +- k== PkSk
13: Server generates new mask: Mnew f- TorK (Tggo((gg
(8global, [(1 o)) -0. .d))
14:
15: // Phase 3: Update Model State
16: m + mpew
New active weights automatically start at init values present un 18: Broadcast updated mask m to all clients
17:
19: return m, w,
25

---

Published in Transactions on Machine Learning Research (12/2025)
A.6.1 Algorithm Extension for OOD adaptation
We formalize our OOD adaptation in Algorithm 2. When OOD clients enter at round Tintro, the server triggers a single mask update. Clients compute saliency scores using the full local model state Wy, effectively "ummasking" the pruned paramcters which retain their original initialization values in memory. This allows the sensitivity analysis to evaluate both the currently active trained weights and the potential utility of inactive connections for the new data.
The aggregated saliency vector Sglobal identifies the optimal subnetwork for the combined distribution. The mask is updated to mnew; activating the new high-saliency connections. Since w, preserves the initialization values for these previously pruned parameters, training simply resumes with the new mask, ensuring a stable optimization landscape without complex restoration steps.
Experiment setup. For proof of concept, we designed a controlled distribution shift experiment using the CIFAR-10 dataset, partitioned into two disjoint tasks: In-Distribution (ID) task consisting of classos 0-7, and an Out-of-Distribution (OOD) task consisting of the held-out classes 8 9 (Ship and Truck). The training timeline was divided into two phases. In the Initial Phase (Rounds 0 0-225), only ID clients participated, ensuring the model had zero exposure to the OOD classes. At Round 225, we introduced the OOD clients and triggered a single execution of our mask discovery algorithm to adapt the global sparse topology to the new data distribution before resuming training.
Alternative Adaptation Strategies and Future Work. While in this preliminary exploration of OOD adaptation we focus on trigger- adaptation for efficiency, other strategies within the SSFL framework are possible. periodic refinement approach could re-evaluate saliency at fixed intervals rather than waiting for specific trigger, offering continuous adaptation at the cost of higher cumulative communication overhead. Alternatively, an incremental growlh strategy could strictly relain the existing mask and only activate additional parameters for OOD data (increasing total density over time). This is possible due to the simplicity and flcxibility of our approach. We leave such explorations for future work.
A.7 Analysis of Warm-up and Mask Discovery at Initialization
A natural question regarding mumiing-a--nnitalizaaie is whether the gradient-based saliency metric defined in Equation (1) is overly sensitive to the random initialization of parameters wn- Intuitively, 'warmup" phase of dense training could stabilize weights and potentially yield more informative gradients. To investigate this in the Federated Learning setting, we conducted a rigorous empirical comparison between our standard SSFL (mask discovery at initialization, t= 0) and Cormup-SSFLL which performs 10 rounds dense federated training before computing saliency and generating the mask.
Empirical Results. We evaluated both methods on CIFAR-10 with ResNet-18 across spectrum of sparsity levels 150%-95%), To ensure fair comparison, both methods were evaluated at the same total after training round (step 500). As shown in Table 6, the performance difference between initialization-base discovery and post-warmup discovery is negligible, with deltas ranging from -0.01% to +0.99%. We leave rigorous analysis of the effect of warm-up for future work.
Table 6: Comparison of SSFT, (mask discovery at initialization) vs. Warmup-SSFL (mask discovery after 10 dense rounds) on CIFAR- 10. All numbers are percentages.
Sparsity
SSFL (At Init)
Warmup-SSFL Difference
95
83.57
83.12
+0.45
90
85.10
84.67
+0.43
80
85.35
86.34
-0.99
70
87.02
87.32
-0.30
50
88.29
88.30
-0.01
26

---

Published in Transactions on Machine Learning Research (12/2025)
Explanation in light of Theoretical Context and Lottery Tickets. Our findings are consistent with the Lottery Ticket Hypothesis literature (Frankle & Carbin, 2019; Frankle et al., 2019; 2020a; Morcos et al., 2019; Chen et al., 2021), specifically the work on Stabilizing the Lottery Ticket Hypothesis (Frankle et al., 2020b) which provides a potential explanation. Frankle et al. (2019) demonstrate that for networks of moderate depth (e.g., ResNet-18) on datasets like CIFAR-10, winning lottery lickets can be successfully identified at initialization (iteration 0). It is primarily for significantly deeper networks or larger datasets (e.g,, ResNet-50 on ImageNet) that "late rewinding" or resetting weights to an early training iteration rather than initialization becomes necessary to find stable subnetworks. Our experimental setting falls within the regime where initialization-bssel pruning is theoretically and empirically robust.
The Effect of Non-IID Data. In the specific context of Federated Learning, we hypothesize that warmup strategies face an additional challenge: chient drift induced by data heterogeneity. During early dense training rounds on non-IID data, local models may drift significantly toward their local distributions. Saliency scores computed after this drift risk capturing paramcters important for local overfitting rather than global generalization. Aggregating these biased scores may cancel out the benefits of weight stabilization. In contrast, the random initialization used in SSFL provides neutral starting point, potentially allowing the aggregated gradient saliency to capture structural importance unbiased by local data quirks.
Conclusion. Given that (1) performance differences are negligible, (2) literature supports initialization-basse pruming for this model scale, and (3) warmup incurs a high communication cost (transmitting dense models for initial rounds), we opted for single-shot discovery at initialization for its optimal efficiency-accuracy trade-off for SSFL. Future work scaling SSFL to very large foundation models may revisit this trade-off, potentially leveraging techniques like warm-up as model capacity grows.
A.8 Experimental settings and further results
The performance of our proposed method is assessed three image classification datasets: CIFAR-10, CIFAR-100 Krizhevsky et al. (2009), and Tiny-lmagenet. We examine two distinct scenarios to simulate non-identical data distributions among federating clients. Following the works of Hsu et al. (2019), we use Dir Partition, where the training data is divided according to a Dirichlet distribution Dir(a) for each client, and the corresponding test data for each client is generated following the same distribution. We take an a value of 0.3 for CIFAR-1O, and 0.2 for both CIFAR-100 and Tiny-Imagenet. Additionally, we conduct an evaluation using a pathological partition setup, as described by Zhang et al. (2020), where each client is randomly allocated limited classes from the total number of classes. Specifically, each client holds 2 classes for CIFAR-10, 10 classes for CIFAR-100, and 20 classes for Tiny-Imagenet in our setup, similar to Dai et al. (2022).
Experiment Details and hyper-paramcters As outlined in Algorithm 1, the client modols are trained on their local data and only during a communication round r, the randomly selected clients in C' share their weights for aggregation. During the local training, we use the SGD optimizer for SSFL and also for all bascline techniques, employing a weighted decay parameter of 0.0005. With the exception of the Ditto method, we maintain constant of 5 local epochs for all methods. However, for Ditto, in order to ensure equitable comparison, each client undertakes 3 epochs for training the local model and 2 epochs for training the global model. Our initial learning rate stands at 0.1 and diminishes by a factor of 0.998 after each round of communication, similar to Dai et al. (2022). Throughout all experiments, we use a batch size of 16 due to the usage of group normalization Wu & He (2018). We execute R 500 global communication rounds for CIFAR-10, CIFAR-100 and Tmag-Im For the experiments in Table I and Table 2, we use a sparsity level s = 50% similar to Dai et al. (2022) for fair comparison. We however, include experiments with increasing sparsity levels up to 95% and compare SSFL with other sparse FL methods, including bascline random-masking on non-IID data, in Figure 2.
For M we use their official implementation Isik et al. (2022) and exlend it to support ResNet architectures and the CIFAR-106 dataset to ensure fair comparisons. For SparsyFed, we likewise rely on the official implementation Guastolla et al. (2025) and re-run their codebase under our general experimental setup, matching our communication rounds (R- = 500) and Dirichlet a parameters. These a values follow the settings commonly adopted in prior work, including Dai et al. (2022), to provide consistency across baselines.
27

---

Published in Transactions on Machine Learning Research (12/2025)
For all baselines, we used the authors' official implementations and recommended hyperparameterss without additional tuning. SSFL. likewise required no hyperparameter tuning beyond setting the target sparsity level, ensuring fair comparison across methods.
Model Initialization All models in our experiments were randomly initialized using the standard Kaiming (He) Normal initialization scheme (He et al., 2015) for convolutional layers, which helps maintain variance stability in deep networks. The final fully connected layer used PyTorch's default Kaiming Uniform initializa- tion. Batch normalization layers were initialized with weights of 1 and biases of O. This random initialization defines the starting state up from which local saliency scores are computed in the first round, ensuring prior task-specific information is encoded in the model weights before the saliency aggregation step. Across 5 independent runs with different random seeds, SSFL exhibited a standard deviation of < 0.4% in final accuracy, demonstrating that the saliency signal extracted at initialization is consistent and robust to the specific random draw of weights.
We describe most of the implementation detail for the baselines in Section We use the official implementaion of DisPFL by Dai et al. (2022) available on their github page and use similar hyper-parameters for both DisPFL and SSFL. FedPM trains sparse random masks instead of training model weights and finds an optimal sparsity by itself Isik et al. (2022). We use the official implementation of FedPM available in their github page for the paper. For fair comparison, we use the CIFAR-10 non-IID split that we use in this work with the a values as described in Appendix A.2. For random-k baselines, we follow a training strategy that is similar to SSFL, and in essense to FedAvg McMahan & Ramage (2017) or DisPFT. Dai et al. (2022) in terms of the weight aggregation mechanism. To establish random-k as a random sparse training, similar to our proposed sparse FL and DisPFL training method we generate a random-k mask each local site and train and aggregate the weights corresponding to that mask after every T steps of training.
Implementation of Baselines:
top-k Top-k is popular choice for distributed training with gradient sparsification Barnes et al. (2020); Lin et al. (2017) and also in variation of federated learning where gradient aggregation instead of weight aggregation takes place. We, however, did not find works that employ top-k sparsification in the weight- averaging scheme of Federated Averaging. As a result, to test this we implemented a top-k baseline, where the top-k weights from each local model at the end of T steps are shared among the selected clients in each communication round and are aggregated. We note that unlike SSFL., DisPFL or random masking, at the end of training top-k does not result in sparse local models with the benefits of sparse models such as faster inference.
Convergence under longer training and more communication rounds We report the convergence plots for SSFL and other methods in Figure 7 (a-b) when trained for a total of R = 500 commuicatiin rounds. As Ditto does not properly converge within 500 rounds, we conduct further experiments for SSFL, DisPFL and Ditto on a much longer R = 800 communication rounds to analyze the convergence of these methods. We remonstrate the convergence plots Figure-7 (c).
(a)
(b)
(c)
Figure Comparison of convergence rates of SSFL with similar federated and sparse foderated lcarning
methods. (a) CIFAR-10 with R 500 rounds, (b) CIFAR-100 with R = 500 rounds, (c) longer training
R = 800 rounds.
28

---

(i) neural network pruning, (ii) sparsity and efficiency in federated learning, (iii) pruning at initialization, and Published in Transactions on Machine Learning Research (12/2025)
A.9 Quality of the discovered mask across heterogeneous clients
Figure 4 (c-d) provides analysis of how the global mask discovered by SSFL transfers across clients with highly imbalanced data availability. Figure 4 (c) shows the test accuracy of local sparse models trained with the global mask, along with the weighted score assigned to each client. Figure 4 (d) shows the corresponding distribution of training and test samples across clients.
Despite substantial differences in the amount of data available at each site, the global mask enables consistently strong performance across the federation. Clients with very limited data still achieve reasonable accuracy, while high-data elients maintain competitive performance. This demonstrates that the aggregated saliency scores produce a global mask that generalizes across diverse clients, rather than overfitting to any single site. Importantly, this analysis highlights that even clients contributing relatively little data can benefit from the shared sparse subnetwork.
A.10 Local Client Accuracy and Class Distribution
We plot the class distribution with Dir(a) on the CIFAR10 dataset with a 0.3 for 10 random clients and their final accuracy in Fig 8 (a). In Fig (b) we plot the top-10 clients in terms of their final test accuracy and in Fig-(c) the bottom 10 clients in terins of final lest accuracy. We notice that, SSFL finds masks that result in consistent local model performance and even in the bottom 10 clients, the performance remains respectable.
U'A ROUFLEEEE NOY
(c)
Figure 8: (a) Class distribution with 00.() for the CIFAR-10 dataset for 10 random clients and their final local test accuracy (b) the top- 10 clients in terms of their final test accuracy (c) the bottom- 10 clients in terms of their final test accuracy
B Extended Literature Review
For completeness, we provide a broader overview of related work, complementing the focused discussion in Scction of the main text. This section is not required to follow our method or experiments but may provide useful background for readers seeking a more complete survey of related areas. We group this discussion into:
(iv) pruning at initialization in federated learning.
Neural Network Pruning Like most areas in deep learning, model pruning has a rich history and is mostly considered to have been explored first in the 90's (Janowsky, 1989; LeCun et al., 1990; Reed, 1993). The central aim of model pruning is to find subnetworks within larger architectures by removing connections. Model pruning is very attractive for number of reasons, especially for real-time applications on resource-constraint edge devices which is often the case in FL and collaborative learning. Pruning large networks can significantly reduce the demands of inference Elsen et al. (2020) or hardware designed to exploit sparsity Cerebras (2019); Pool et al. (2021). More recently the lottery ticket hypothesis was proposod which predicts the existence of subnetworks of initializations within dense networks, which when trained in isolation from scratch can match in accuracy of fully trained dense network Frankle & Carbin (2019).
29

---

Published in Transactions on Machine Learning Research (12/2025)
This rejuvenated the field of sparse deep learning Renda et al. (2020); Chen et al. (2020) and more recently the interest spilled over into sparse reinforcement learning (RI) as well Arnob et al. (2024; 2021); Sokar et al. (2021). Pruning in deep learning can broadly be classified into three categories: techniques that induce sparsity before training and at initialization Lee et al. (2018); Wang et al. (2020); Tanaka et al. (2020), during training, including methods based on magnitude pruning Zhu & Gupta (2018), weight transformations Ma et al. (2019), regularization-bant basod approaches Yang et al. (2019), explicit sparse projections Ohib et al. (2022); Ohib (2023); Ohib et al. of the matrices and post-training Han et al. (2015); Frankle et al. (2021).
The advent of large language models (LLMs) has further amplified interest in pruning and sparsity techniques, as these models often contain billions of parameters that pose significant computational and memory challenges for deployment. Recent work has explored various pruming strategies for LLMs, including maguitude-based pruning Frantar& & Alistarh (2023), structured pruming to remove entire attention heads or layers Ma et al. (2023), and semi-structured sparsity patterns that balance compression with hardware efficiency Sun et al. (2023). Additionally, techniques such as quantization awaae pruning and knowledge distillation have been combined with sparsity to achieve extreme compression ratios while mainlaining model performance Xiao et al. (2023); Kurtić et al. (2023). Sparsity has also been leveraged in eaffiiter-efficient fine-tuming through sparse adapters Arnob et al. (2025), low-rank adaptations Hu et al. (2022); Zhang et al. (2023), and mixture- of-experts architectures that activate only subsets of parameters per input Fedus et al. (2022): Jiang et al. (2024). These advances demonstrate that pruning remains crucial for making state-of-theeart models practical for resource-constrained environments, extending from traditional computer vision tasks to modern natural language processing applications.
Sparsity, pruning and efficiency in FL Federated Learning (F'L) has evolved significantly since its inception, introducing a myriad of techniques to enhance efficiency, communication, and computation. Interest in efficiency has increased further with the proposal of the Lottery Ticket. Hypothesis (I.T)) by Frankle & Carbin (2019). However, identifying these tickets has traditionally been challenging. In the original work, the strategy employed to extract these subnetworks was to iteratively train and prune the network until the target sparsity was attained. However, this process is unsustainable in the FL setting and would in contrast make the training even more expensive due to the significant increase in training duration. Variations of this idea have been employed in the FL setting, as demonstrated by Li et al. (2020); Jiang et al. (2022); Seo et al. (2021); Liu et. al. (2021) and Babakniya el al. (2022). However, many of them either suffer from the same issue of iterative pruning and retraining which is extremely costly or decline in performance when this train, prune and retrain cycle is skippod.
Pruning and sparsity are highly advantageous in federated learning (FL.), especially since many applications involve clients with limited resources. As sparse models can be substantially efficient during inference (Li et al., 2016), training sparse local models in resource constrained local sites could be an effective approach. In FL setting, pruning has been explored with mixed success, often employing a range of heuristics. Works, such as by Munir et al. (2021) focused on pruning resouree-constrainee clients, Yu et al. (2021) on using gated dynamic sparsity, Liu et al. (2021) performs a pre-training on the clients to get mask information, and Jiang et al. (2022) relies on an initial mask selected at a particular client, followed by a FedAvg-like algorithm that performs mask readjustment. every AR rounds. However, they either do not leverage sparsity fully for communication efficicncy, suffer from performance degradation or is not designed for the challenging, realistic non-IID setting.
Works by Dai et al. (2022) and Bibikar et al. (2022) utilize dynamic sparsity similar to Evei et al. (2020a) and extends it to the FL regime, however, they both start with random masks as an initialization. A recent focus has also been placed on random masking and optimizing the mask Setayesh et al. (2022) or training the mask itself instead of training the weights at all Isik et al. (2022). Furthermore, in light of the recent surge in the development of large language models (LLMs), the exploration of federated learning in this domain is taking place, although relatively recently Fan et al. (2023); Yu et al. (2023). Investigating sparsity and efficiency within this context presents a fascinating and promising avenue for research.
Pruning at Initialization The work on the lottery ticket hypothesis (LTH), which showed that from early in training and often at initialization, there exist subnetworks that can be trained in isolation to full accuracy,
30

---

Published in Transactions on Machine Learning Research (12/2025)
opened up the prospect for pruning at initialization (PaI). Several methods have been proposed for pruming at initialization such as SNTP (Lee et al., 2018), which aims to prune weights that are least important for the loss, GraSP (Wang et al., 2020), which prunes weights aiming to preserve gradient flow and SynFlow (Tanaka et al., 2020) aims to iteratively prune weights with the lowest "synaptic strengths". These methods have been later studied in depth by Frankle et al. (2021), where they provide interesting insights about the efficacy of these methods, including the observation that SNTP Lee al. (2018), which variation of the gradient based connection saliency criterion (Mozer & Smolensky, 1988), consistently performs well. Hence, choose gradient based connection saliency as the basis for our method.
Frankle et al. (2021) found that PaI methods tend to prune networks at the layer level rather than the connection level in single-node deep learning scenarios. In Section 4, we explore the applicability of this observation to non-IID federated learning settings
Pruning at initialization in FL We did not discover any study that directly employs connection importance at initialization in the Federatod Learning (FL) setting. Nevertheless, research such as Jiang et al. (2022) incorporated gradient-based importance criterion, SNIP Lee et al. (2018); De Jorge et al. (2020), within their comparative benchmarks, but calculating such scores only the local data of particular client. site in the FL setting, resulting in subpar performance. Similarly, Huang et al. (2022) presents connection saliency in their benchmark but assumes that to calculate meaningful parameter importance scores, the server has access to a public dataset on which such saliency scores can be calculated. However, that is in general unlikely to happen and would lead to privacy concerns if such constraints were enforced.
In this work, we demonstrate that possible to find sparse masks or sub-networks through gradient basod connection saliency measures at initialization, considering the distribution of training data at local client sites. Since, to the best of our knowledge we did not find any comparable pruning at initialization methods in the non-IID FL setting, compare our method to approaches such as Dai et al. (2022) that results in sparse local models and similar sparse and dense baselines in Section 4. A variation of our work has also been applied in the domain of federated distributed neuroimaging (Thapaliya et al., 2024), where gradient averaging, first demonstrated in (Ohib et al., 2023) in this sctup, was used for the FL process instead of model averaging.
31


Reviews:
[{"review_id": "tyfBKXyeyI", "reviewer": "Reviewer_pA7V", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work investigates availability of a sparse subnetwork at initialization for  communication-efficient federated training on non-i.i.d. data. The identification of a shared sparse mask is done through  local computation of 'parameter saliency  scores' and their 'one-shot' aggregation at the server,-  to begin the training process. The resulting approach is demonstrated to be communication-efficient, privacy-compliant, and outperforms state-of-the-art sparse FL baselines.\n\nStrengths:\n* The paper is well-motivated, easy to follow, and on a topic is of significant interest.\n* The idea of 'one-time sparse mask' discovery for instantiation of the learning process is interesting.\n* The method of identifying salient parameters, i.e., saliency-based pruning, is based on established approaches on neural network pruning (see [R1, R2]), and the results shows impressive performance gain as compared to existing works.\n\n\nWeakness:\n* The novelty of this work appears limited, as compared to [R1, R2] - the SNIP paper. This is one of the major concern I have had while going through this work - and the others.\n* Avoiding iterative interactions is probably a merit, however, the proposed method appears not competent enough to adapt with changing data distributions, unlike the related works - e.g., SparsyFed (Guastella et al., 2025)); hence, limiting the applicability of the work.\n* The computational cost and complexity analysis is missing.\n* Random baseline is missing in Table 3. Does it not make sense to include SNIP within FL setting as an strategy and indicate the performance as a baseline.\n\n[R1] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on\nconnection sensitivity. arXiv preprint arXiv:1810.02340, 2018.\n[R2] De Jorge P, Sanyal A, Behl HS, Torr PH, Rogez G, Dokania PK. Progressive skeletonization: Trimming more fat from a network at initialization. arXiv preprint arXiv:2006.09081. 2020 Jun 16.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims made appear accurate, within the experimental setting and results presented in the paper, and equally well-situated with the related works. The proposed approach is build on, particularly [R1] (and [R2]), both cited in the paper.\n\n[R1] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on\nconnection sensitivity. arXiv preprint arXiv:1810.02340, 2018.\n\n[R2] De Jorge P, Sanyal A, Behl HS, Torr PH, Rogez G, Dokania PK. Progressive skeletonization: Trimming more fat from a network at initialization. arXiv preprint arXiv:2006.09081. 2020 Jun 16.", "audience_yn": "Yes", "audience_explanation": "The works provides single-round sparse federated learning approach that offers better training efficiency as compared with the existing approaches: enabling low communication overhead and accelerated wall-clock communication.", "requested_changes": "* I think the authors should better justify the novelty of this work as compared to the related works.\n* Please discusses, what are the considerations and implications of assumption made with a single global mask at initialization to align all updates in the same sparse subspace: \"clients remain in a common subspace throughout training\".\n* The claim regarding the use of only a single minibatch per client should be justified and validated for performance guarantees. \n* What would be the computational cost and complexity of the proposed approach, as compared with the related works. Can the framework handle system-level heterogeneity, i.e., difference in compute abilities of the clients?\n* Discuss implications for using static mask and how to handle changing data distributions. I believe experimenting on this setting would bring both novelty and contributions to the work.\n* Fig. 3 (b) is missing other baselines for fair assessment.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "D0saTgUZbg", "reviewer": "Reviewer_yNpy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors propose Salient Sparse Federated Learning (SSFL), the first single-shot sparse federated learning framework that discovers a globally shared sparse subnetwork at initialization using only clients’ private data. SSFL introduces a one-round saliency aggregation mechanism that efficiently combines local gradient-based importance scores to form a global sparse mask, achieving both communication efficiency and privacy preservation.", "claims_and_evidence_yn": "No", "claims_explanation": "The proposed method is relatively straightforward and intuitive; however, I have two major concerns regarding its validity.\n\n(1) In the paper, the mask for parameter selection is determined by the saliency score defined in Equation (1). Since the saliency score depends on the parameter gradients, which continuously change throughout the optimization process, the saliency score itself also varies during training. The proposed method, however, uses a fixed mask computed only from the saliency scores at the initialization point. This design appears unreasonable, as a more appropriate approach would be to dynamically update the mask according to the evolving saliency scores during optimization.\n\n(2) Furthermore, when computing the saliency scores at initialization, the authors use only a single minibatch of data. Because minibatch selection is inherently stochastic, the computed saliency scores at initialization are also random. Consequently, the mask derived from these scores may be unreliable.", "audience_yn": "Yes", "audience_explanation": "Efficient federated learning is indeed an important research direction, and the problem addressed in this paper is highly relevant. However, the proposed method appears to be not particularly well-justified or theoretically sound.", "requested_changes": "See the two questions above regarding the soundness and rationality of the proposed method.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "49dpb5SEbs", "reviewer": "Reviewer_YzoX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Salient Sparse Federated Learning (SSFL), which is a framework for deciding how to sparsify models distributed across different clients with non-IID and differently balanced data in an FL setting. SSFL implements static sparsity in a one-shot manner by estimating per-parameter gradient saliency based on one minibatch from each client at model initialization, and then aggregating saliency scores from all clients as an average score weighted by the size of the local dataset that a given score was calculated from. Then, a TopK selection is performed based on a selected sparsity level to select the K parameters with the largest saliency scores. A mask is then produced to null out all other parameters, and this mask is communicated to all local models to be used for all subsequent training.\n\nIn consequence, dynamic updates of the masks are not required, which reduces communication costs between clients. Because all local models share a global mask, global model updates are straightforward since all local models are structurally aligned, and furthermore makes the global updates meaningful since all models then work in the same sparse subspaces. SSFL also outperforms other sparse FL methods with higher accuracy and less data being communicated, at different levels of sparsity. An experiment emulating real-life FL conditions also shows a large wall-clock time saving compared to dense FL training.\n\nStrengths:\n- The paper is well-written and explains relevant concepts, details of the SSFL framework and experimental details very well.\n- It points out the challenges in the field that it adresses (system heterogeneity and statistical heterogeneity), outlines how existing methods address the same challenges, which issues these methods face (e.g., multiple rounds of communication with dynamic sparsity, public proxy datasets violating privacy constraints of many FL applications), and how SSFL addresses the same issues.\n- The method is sound, and elegant in its simplicity. The appendix also explains the choices of saliency criterion and non-IID data partition in great detail.\n- The experiments are described clearly and in detail, and the results clearly point to that the claimed contributions were achieved. An extensive amount of relevant experiments were performed, which substantiate the authors' claims.\n\nWeaknesses:\n- My main criticism of the paper regards the size of the model used for the main experiments and results (subsection 4.1), where only ResNet18 is used for all tests. While the same model was used in related papers and it is therefore important to see SSFL results with the same model for benchmarking reference, it would be natural to examine how SSFL performance scales up as model size increases, for instance with a larger ResNet model or a ViT model. Especially considering that a major contributing factor to the added efficiency of SSFL compared to other methods lies in reduced communication costs, the effect of this could possibly be even larger for larger-scale models, which would be interesting to examine. If it is infeasible to perform experiments on this prior to the rebuttal deadline, I think not having measured the degree of scalability the SSFL framework has should be cited in limitations/future work.\n- Regarding the method; since saliency estimation is performed after parameter initialization for a given local model, would not the saliency metric potentially be sensitive to however each parameter is initialized? If so, it would be interesting to see the effect of waiting a little while before performing saliency estimation and aggregation, for instance a warmup phase consisting of a set number of iterations. Also with regards to this, I could not find any details of how the ResNet18 model used in the experiments was initialized, beyond this sentence in A.1: \"The fundamental goal of pruning at initialization is to identify the most important parameters in a randomly initialized network, w0, without performing any training\", which only suggests that the parameters were randomly initialized without explicitly confirming it. The element of randomness in parameter initialization is what makes me think it would be good to measure how both the already included performance metric measurements and also the uncertainty of these measurements would be affected if SSFL was delayed for some iterations.\n- It would be good to include confidence intervals for the table values, considering they were measured over five runs. This is also interesting wrt. my other criticism above.\n\nLanguage comments/typos:\n- Title of Appendix A.2 says \"parition\" instead of \"partition\".\n- Under 3.3, paragraph \"Pathological Partition\", the formulation \"...where a limited number of classes are only assigned to each client at random from the total number of classes...\" was unclear in my opinion. If I understood it correctly, it could be rewritten as \"...where each client is randomly assigned only a limited number of classes from the total number of classes\", etc.\n- In the list of contributions, point 2: \"a globally representative importance scores\" should have the \"a\" removed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper demonstrates and explains that it indeed achieves single-shot model pruning in an FL setting that avoids auxiliary datasets, iterative pruning and additional hyperparameters.\n\nThe SSFL method addresses the challenge of data heterogeneity, while adhering to FL principles of client privacy.\n\nThe presented experimental results clearly demonstrate better results than other state-of-the-art sparse FL methods, in the non-IID setting that the paper addresses.\n\nThe presented results also demonstrate a faster wall-clock time when applied on a real-world FL system.\n\nGenerally in the paper, other claims and arguments are well-argued for and substantiated by support in related work and experimental results.", "audience_yn": "Yes", "audience_explanation": "Anyone having interest in federated learning and sparse FL in particular would naturally be interested. The same goes for any audience interested in model pruning, distributed training, and anyone interested in efficient model training in general.", "requested_changes": "While I consider this a strong paper as-is, my requested changes build upon my criticisms written above, and would in my opinion strengthen the paper further. They are:\n\n1) Perform experiments with larger models with more parameters, such as a larger ResNet model or a ViT. It would substantiate the generality of the effect that reduced communication has on training efficiency in the sparse FL setting. If infeasible, cite this as a limitation and/or future work.\n\n2) Please address how (random) parameter initialization affects pre-training saliency estimation. My proposition is to try and run SSFL after a warmup phase and measure experimentally how performance measurements and their uncertainty is affected.\n\n3) Add confidence intervals for the values in the result tables, this may also provide an answer for 2) to some extent.", "broader_impact_concerns": "There already is a sufficiently good Broader Impact Statement in the paper, no further change is needed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)