Title: SpaFL: Communication-Efficient Federated Learning With Sparse Models And Low Computational Overhead
Venue: NeurIPS 2024
Accepted: yes
Keywords: Federated Learning

Communication and Computation Efficiency

Pruning
Abstract: The large communication and computation overhead of federated learning (FL) is one of the main challenges facing its practical deployment over resource-constrained clients and systems. In this work, SpaFL: a communication-efficient FL framework is proposed to optimize sparse model structures  with low computational overhead. In SpaFL, a trainable threshold is defined for each filter/neuron to prune its all connected 
parameters, thereby leading to structured sparsity. To optimize the pruning process itself,  only thresholds are communicated between a server and clients instead of parameters, thereby learning how to prune. Further, global thresholds are used to update model parameters by extracting aggregated parameter importance. The generalization bound of SpaFL is also derived, thereby proving key insights on the relation between sparsity and performance. Experimental results show that SpaFL improves accuracy while requiring much less communication and computing resources compared to sparse baselines. The code is available at https://github.com/news-vt/SpaFL_NeruIPS_2024

SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low Computational Overhead

Abstract

The large communication and computation overhead of federated learning (FL) is one of the main challenges facing its practical deployment over resourceconstrained clients and systems. In this work, SpaFL: a communication-efficient FL framework is proposed to optimize sparse model structures with low computational overhead. In SpaFL, a trainable threshold is defined for each filter/neuron to prune its all connected parameters, thereby leading to structured sparsity. To optimize the pruning process itself, only thresholds are communicated between a server and clients instead of parameters, thereby learning how to prune. Further, global thresholds are used to update model parameters by extracting aggregated parameter importance. The generalization bound of SpaFL is also derived, thereby proving key insights on the relation between sparsity and performance. Experimental results show that SpaFL improves accuracy while requiring much less communication and computing resources compared to sparse baselines. The code is available at https://github.com/news-vt/SpaFL_NeruIPS_2024

1 Introduction

Federated learning (FL) is a distributed machine learning framework in which clients collaborate to train a machine learning (ML) model without sharing private data [1]. In FL, clients perform multiple epochs of local training using their own datasets and communicate model updates with a server. Different from a classical, centralized ML, FL systems are typically deployed on edge devices such as mobile or Internet of Things (IoT) devices, which have limited computing and communication resources. However, current ML models are typically too large and complex to be trained and deployed for inference by edge devices. Moreover, large model sizes can induce significant FL communication costs on both devices and communication networks. Hence, the practical deployment of FL over resource-constrained devices and systems requires optimized computation and communication costs for both edge devices and communication networks. This has motivated lines of research focused on reducing communication overhead in FL [2, 3], training sparse models in FL [4, 5, 6, 7, 8, 9], and optimizing model architectures to find a compact model for inference [10, 11, 12]. The works in [2, 3] proposed training algorithms such as quantization, gradient compression, and transmitting the subset of models in order to reduce the communication costs of FL. However, the associated computational overhead of these existing algorithms remains high since devices have to train a dense model. In [4, 5, 6, 7, 8, 9], FL algorithms in which devices train and communicate sparse models are proposed. However, the works in [4, 5] used unstructured pruning, which is difficult to gain the computation efficiency in practice. Moreover, the computation

and communication overhead can still be large if model sparsity is not high. In [6, 7, 8, 9], the authors investigated the structured sparsity, however, the solutions therein either fixed the channel sparsity patterns for clients or did not optimize the pruning process. Furthermore, the FL approaches of [10, 11, 12] can significantly increase computation resource usage by training multiple models for resource-constrained devices. Clearly, despite a surge of literature on sparsity in FL, there is still a need to develop new FL algorithms that can find sparse model structures with optimized communication efficiency and low computational overhead to operate on resource-constrained devices.

The main contribution of this paper is SpaFL: a communication-efficient FL framework for optimizing sparse models with low computational overhead achieved by performing structured pruning through trainable thresholds. Here, a trainable threshold is defined for each filter/neuron to prune all of its connected parameters. To optimize the pruning process, only thresholds are communicated between clients and the FL server. Hence, clients can learn how to prune their model from global thresholds and can significantly reduce communication costs. Since parameters are not communicated, the clients' parameters and sparse model structures will remain personalized while only global thresholds are shared. We show that global thresholds can capture the aggregated parameter importance of clients. We further update the clients' model parameters by extracting aggregated parameter importance from global thresholds to improve performance. We analyze the generalization ability of SpaFL and provide insights on the relation between sparsity and performance. We summarize our contributions as follows:

We propose a new communication-efficient FL framework called SpaFL, in which clients optimize their sparse model structures with low computing costs through trainable thresholds.

We show how SpaFL can significantly reduce communication overhead for both clients and the server by only exchanging thresholds, the number of which is less than two orders of magnitude smaller than the number of model parameters.

We provide the generalization performance of SpaFL. Moreover, the impact of sharing thresholds on the model performance is theoretically and experimentally analyzed.

Experimental results demonstrate the performance, computation costs, and communication efficiency of SpaFL compared with both dense and sparse baselines. For instance, the results show that SpaFL uses only 0.17% of communication and 12.0% of computation resources compared to a dense baseline FedAvg while improving accuracy. Additionally, SpaFL improves accuracy by 2.92% compared to a sparse baseline while consuming only 0.35% of this baseline's communication resources, and only 24% of its computing resources.

2 Background and Related Work

2.1 Federated Learning

Distributed machine learning has consistently progressed and achieved success. However, it mostly focuses on training with independent and identically distributed (i.i.d.) data [13, 14]. The FL frameworks along with the FedAvg [1] enables clients to collaboratively train while preserving data privacy without data sharing. Due to privacy constraints and individual preferences, FL clients often collect non-iid data. As such, data can exhibit differences and imbalances in distribution across clients. This variability poses significant challenges in achieving efficient convergence. For a more detailed literature review, we refer to [15, 16]. Although most of state-of-the-art FL methods are effective in mitigating data heterogeneity, they often neglect the computational and communication costs involved in the training process.

2.2 Training and Finding Sparse Models in FL

To reduce the computation and communication overhead of complex ML models during training, the idea of embedding FL algorithms with pruning has recently attracted attention. In [4, 5, 6, 7, 8, 9, 17, 18, 19, 20, 21, 22, 23, 24, 25], the clients train sparse models and communicate sparse model parameters to reduce computation and communication overhead. To improve the aggregation phase with sparse models, the works in [17, 20, 21] perform averaging only between overlapping parameters to avoid information dilution by excluding zero value parameters. The authors in [18] obtained a sparse model by selecting a particular client to prune an initial dense model and then

Global thresholds

t(t + 1) =

Tk(t)

Thresholds

Thresholds

T1(t)

TN(t)

Server

Structured sparse model

Data(

Client 1

Client N Data

Structured sparse model

W1(t)

WN(t)

performed training in a similar way to FedAvg. In [4, 24], the authors presented binary masks adjustment strategy to improve the performance of sparse models and communication efficiency. The work in [25] progressively pruned a dense model for sparsification and analyzed its convergence. In [19, 22], the clients optimized personalized sparse models by exchanging lottery tickets [26] at every communication round. The work in [5] obtained personalized sparse models by l 1 norms constraints and the correlation between local and global models. In [8], the authors proposed dual pruning scheme for both local and global models to reduce the communication costs. The FL framework of [23] allows clients to train personalized sparse models in a decentralized setting without a central server. Although these works [17, 18, 4, 24, 25, 19, 22, 5, 23] adopted sparse models during training, they used unstructured pruning, which is difficult to improve the computation efficiency in practice. Meanwhile, with structured sparsity, the authors [7] proposed a training scheme that allows clients to train smaller submodels of a global model. In [9], clients train set of submodels with fixed channel sparsity patterns depending on their computing capabilities. The work in [6] studied structured sparsity by adjusting clients' channel activation probabilities. However, the works in [7, 9] fixed sparsity patterns and did not optimize sparse model structures. Although [6] optimized channel activation probabilities, the communication cost of downlink still remains high as a server broadcasts whole parameters. Similar to our work, in [27, 28], only binary masks are communicated and optimized by training auxiliary variables to learn sparse model structures. However, the work in [27] approximated binarization step using a sigmoid function during forward propagation. In [28], the downlink communication costs remained the same as that of FedAvg. In [10, 11, 29], clients perform neural-architecture-search by training multiple models to find optimized and sparse models to improve computational and memory efficiency at inference phase. However, in practice, clients often have limited resources to support the computationally intensive architecture search process [30]. Therefore, most prior works either adopted unstructured pruning or they still required extensive computing and communication costs for finding optimal sparse models. In contrast to the prior art, in the proposed SpaFL framework, we find sparse model structures with structured sparsity by optimizing and communicating trainable thresholds for filter/neurons.

3 SpaFL Algorithm

In this section, we first present the proposed pruning scheme for structured sparsity and formulate our FL problem to find optimal sparse models. Then, we present SpaFL to solve the proposed problem with low computation and communication overhead.

3.1 Structured Pruning with Trainable Thresholds

We define a trainable threshold for each neuron in linear layers or for each filter in convolutional layers. The neural network of client k will consist of L layers as { W 1 k , . . . , W L k } . For parameters W l k ∈ R n l out × n l in in a linear layer l , we define trainable thresholds τ l ∈ R n l out for output neurons. If it

is a convolutional layer W l k ∈ R n l out × c l in × k l × h l , where c l in is the number of input channels and k l × h l are the kernel sizes, we can change W l k as W l k ∈ R n l out × n l in with n l in = c l in × k l × h l . Similarly, we can define the corresponding thresholds τ l ∈ R n l out for filters in that layer. Then, for each client k , we define a set of total thresholds τ = { τ 1 , . . . , τ L } . Note that the number of these additional thresholds will be at most 1% of the number of model parameters d .

For threshold τ l i of filter/neuron i in layer l , we compare the average magnitude of its connected parameters µ l k,i = 1 /n l in ∑ n l in j =1 | w l k,ij | to its value τ l i . If µ l k,i < τ l i , we prune all connected parameters to this filter/neuron. Hence, our pruning can induce structured sparsity unlike [31]. Thus, we do not need to compute the gradients of parameters in a pruned filter/neuron [32] during backpropagation. We can obtain a binary mask p l k for W l k , as follows

where S ( · ) is a unit step function. Hence, we can obtain the binary masks { p 1 k , . . . , p L k } by performing (1) at each layer. To facilitate the pruning, we constrain the parameters and thresholds to be within [ -1 , 1] and [0 , 1] , respectively [31]. For simplicity, we unroll { W 1 k , . . . , W L k } and { p 1 k , . . . , p L k } to w k ∈ R d and p k ∈ R d , respectively as done in [33]. Thresholds represent the importance of their connected parameters (see more details in Section 3.3.1). Hence, clients can know which filter/neuron is important by training thresholds, thereby optimizing sparse model structures. Then, the key question becomes: Can clients benefit by collaborating to optimize shared thresholds in order to find optimal sparse models? We partially answer this question in Table 1. Following the same configurations in Section 5, clients with non-iid datasets only train and communicate thresholds τ while freezing model parameters.

We can see that learning sparse structures can improve the performance even without training parameters. This also corroborates the result of [28]. Motivated by this observation, we aim to find optimal sparse models of clients in an FL setting by communicating only thresholds in order to reduce the communication costs in both clients and server sides while keeping parameters locally. The communication cost will decrease drastically because the number of thresholds will be at most 1% of the number of model parameters d . Essentially, we optimize the sparse models of clients with small computing and communication resources by communicating thresholds.

3.2 Problem Formulation

We aim to optimize each client's model parameters and sparse model structures jointly in a personalized FL setting by only communicating thresholds. This can be formulated as the following optimization problem:

where ˜ w k = w k ⊙ p k ( τ ) is a pruned model, F k ( · ) is a empirical risk associated with local data of client k , L is a loss function, D k is the number of data samples, { x , y } is an input-label pair, w k captures the model parameters, and ⊙ is the Hadamard product. If an element of p k ( τ ) is zero, then the corresponding parameter of w k will be pruned. Our goal is to obtain the optimal w k and τ for each client in order to reduce the computation and communication overhead during training. However, solving (2) is not trivial because w k and τ are highly correlated. Moreover, structured sparsity can induce a large performance drop due to coarse-grained sparsity patterns compared to unstructured pruning [34].

3.3 Algorithm Overview

Wenow describe the proposed algorithm, SpaFL, that can solve (2) while maintaining communicationefficiency with low computational cost. In SpaFL, every client jointly optimizes its personalized sparse model structure and model parameters with trainable thresholds, which can be used to prune filters/neurons. To save communication resources, only thresholds will be aggregated at a server to generate global thresholds for the next round. Here, global thresholds can represent the aggregated parameter importance of clients. Hence, at the beginning of each round, every client extracts the aggregated parameter importance from the global thresholds so as to update its model parameters. The overall algorithm is illustrated in Fig 1. and summarized in Algorithm 1.

3.3.1 Local Training for Parameters and Thresholds

At each round, a server samples a set of clients S t such that |S t | = K for local training. For given global thresholds τ ( t ) at round t , client k ∈ S t generates a binary mask p k ( τ ( t )) using (1). Subsequently, it obtains the sparse model ˜ w k ( t ) = w k ( t ) ⊙ p k ( τ ( t )) . To improve the communication efficiency, each sampled client performs E epochs using mini-batch stochastic gradient to update parameters and thresholds as follows:

where g k ( ˜ w e k ( t )) = ∇ ˜ w e k F k ( ˜ w e k ( t ) , τ ( t ); ξ e k ( t )) , h k ( ˜ w e k ( t )) = ∇ τ F k ( ˜ w e k ( t ) , τ ( t ); ξ e k ( t )) with a mini-batch ξ and η ( t ) is a learning rate. Parameters of unpruned filter/neurons and thresholds will be jointly updated via backpropagation. To enforce sparsity, we add a regularization term R ( t ) to (4) in order to penalize small threshold values. To this end, client k first calculates the following sparsity regularization term R ( t ) = ∑ L l =1 ∑ n l out i =1 exp( -τ i ) . Then, the loss function can be rewritten as:

where 0 ≤ α ≤ 1 is the coefficient that controls R ( t ) . From (5), we can give thresholds τ ( t ) performance feedback on the current sparse model while also progressively increasing τ ( t ) through the sparsity regularization term R ( t ) [31]. From (5), client k then updates the received global thresholds τ ( t ) via backpropagation as follows

After local training, each client k ∈ S t , transmits the updated thresholds τ k ( t ) to the server. Here, the communication overhead will be less than one percent of that of transmitting the entire parameters. Subsequently, the server performs aggregation and broadcasts new global thresholds, i.e.,

Here, in SpaFL, clients communicate only thresholds. Then, what will clients learn from sharing trained thresholds? Next, we show that thresholds represent the importance of their associated filter/neurons.

3.3.2 Learning Parameter Importance From Thresholds

Clients can know which filter/neurons are important by sharing trained thresholds. For the threshold of filter/neuron i at layer l of client k , its gradient can be written as below

where Q e,l k,i ( t ) = µ e k,i ( t ) -τ e,l k,i ( t ) in (1), (8) is from the definition of pruned parameters in (2) and the unit step function S ( · ) , and (9) is from the identity straight-through estimator [35] to approximate the gradient of the step functions in (8).

From (9), we can see that threshold τ e,l k,i corresponds to the importance of its connected parameters w e,l k,ij , 1 ≤ j ≤ n l in , in its filter/neuron. This is because the importance of a parameter w l ij can be estimated by [36]

where F ( w , τ ; w l ij = 0) is the loss function when w l ij is masked and the approximation is obtained from the first Taylor expansion at w l ij = 0 . Therefore, if connected parameters were important, the sign of (10) of those parameters will be negative, and the corresponding threshold will decrease as in (9). Otherwise, the threshold will be increased to enforce sparsity. Hence, prematurely pruned parameters will be automatically recovered via a joint optimization of τ and w .

3.3.3 Extracting Parameter Importance from Global Thresholds

Since thresholds represent the importance of the connected parameters at each filter/neuron, clients can learn how to prune their parameters from the global thresholds. Moreover, the difference between two consecutive global thresholds ∆ τ ( t ) = τ ( t + 1) -τ ( t ) captures the history of aggregated parameter importance, which can be further used to improve model performance. For instance, from (10), if ∆ τ l i ( t ) < 0 , then the parameters connected to threshold i in layer l were globally important. If ∆ τ l i ( t ) ≥ 0 , then the connected parameters were globally less important. Hence, from ∆ τ ( t ) , clients can deduce which parameter is globally important or not and further update their model parameters. After generating new global thresholds τ ( t +1) , the server broadcasts τ ( t +1) to client k ∈ S t +1 , and then clients calculate ∆ τ ( t ) = τ ( t +1) -τ ( t ) .

We then present how clients can update their model parameters from ∆ τ ( t ) . For given ∆ τ ( t ) , we need to decide on the: 1) update direction and 2) update amount. Clients can know the update direction of parameters by considering ∆ τ ( t ) and the dominant sign of parameters connected to each threshold. For simplicity, assume that each parameter has a threshold. Then, the gradient of the thresholds in (9) can be rewritten as follows:

The gradient of the loss F k ( ˜ w k ( t ) , τ ( t )) with respect to the whole parameters w k ( t ) is given by

From (11) and (12), the gradient direction of a parameter w is opposite of that of its connected threshold if w > 0 . Otherwise, both the threshold and the parameter have the same gradient direction.

Algorithm 1: SpaFL

Hence, we can deduce the following: If w > 0 , the gradient direction of w and the sign of ∆ τ will have the same sign; otherwise, the gradient direction of w and the sign of ∆ τ are opposite. In SpaFL, each threshold has multiple connected parameters to its filter/neuron. As such, we decide the update direction of connected parameters by finding the dominant sign among them. To this end, we simply add the connected parameters of each threshold. For instance, consider threshold i in layer l of client k , if ∑ n l in j =1 w l k,ij ( t ) > 0 , then the gradient direction of the connected parameters will be the same as the sign of ∆ τ l i ( t ) . Otherwise, it is the opposite of the sign of ∆ τ l i ( t ) . Thus, the update direction can be simply expressed with a XOR operation between the sign of ∆ τ l i ( t ) and the sign of connected parameters sum. Next, we decide how much a parameter should be updated. From (11) and (12), we can see that a threshold and a parameter have the same magnitude for their gradients. Hence, we simply divide ∆ τ l i ( t ) by the number of connected parameters n l in . We finally provide the update equation using ∆ τ ( t ) as follows

where sign ( · ) is a sign function. This parameter update corresponds to line 7 in Algorithm 1. Note that this additional parameter update is not computationally intensive because it happens only once before local training. We also provide the number of used FLOPs during training with inclusion of this operation in Section 5.

4 Theoretical Analysis of SpaFL

We now present our generalization analysis of SpaFL. For the empirical risk ˆ R = 1 N ∑ N k =1 1 D k ∑ D k i =1 L ( ˜ w k , τ ; z i ) , we consider the expected risk R = 1 N ∑ N k =1 E z k ∼D k L ( ˜ w k , τ ; z k ) , where L is a loss function and z is an input-output pair. Suppose ρ k is the ratio of remaining model parameters of client k and ¯ ρ is the average model density across clients. Then, for the hypothesis A ( D ) with global thresholds τ from Algorithm 1 on the joint training dataset D = ∪ N k =1 D k with ¯ ρ , we have the following generalization bound as follows:

Theorem 1. For the loss function ||L|| ∞ ≤ 1 , the training data size D ≥ 2 ϵ ′ 2 ln ( 16 exp( -ϵ ′ δ ′ ) ) and the total number of communication rounds T , we have

where ξ is the size of a mini-batch, σ is the variance of Gaussian noise, and M g is the maximum diameter of thresholds' gradients (11) . The proof and the definition of δ are provided in the Appendix 1.2 and (12), respectively.

From Theorem 1, we can see that, as the average model density ¯ ρ decreases, the generalization bounds becomes smaller, thereby achieving better generalization performance. This is because ϵ ′ and ˜ ϵ decrease as the average model density ¯ ρ decreases. Hence, SpaFL can improve the generalization performance with sparse models by optimizing and sharing global thresholds.

5 Experiments

We now present experimental results to demonstrate the performance, computation costs and communication efficiency of SpaFL. Implementation details are provided in the Supplementary document.

5.1 Experiments Configuration

We conduct experiments on three image classification datasets: FMNIST [37], CIFAR-10, and CIFAR-100 [38] datasets with NVIDA A100 GPUs. To distribute datasets in a non-iid fashion, we use Dirichlet (0.2) for FMNIST and Dirichlet (0.1) for CIFAR-10 and CIFAR-100 datasets as done in [39] with N = 100 clients. We set the total communication round T = 500 and 1500 for FMNIST/CIFAR10 and CIFAR100, respectively. At each round, we randomly sample K = 10 clients. Unless stated otherwise, we average all the results over at least 10 different random seeds. We also calculate the best accuracy by averaging each client's performance on its test dataset. For FMNIST dataset, we use the Lenet-5-Caffe. For the Lenet model, we set η ( t ) = 0 . 001 , E = 5 , α = 0 . 002 , and a batch size to be 64. For CIFAR-10 dataset, we use a convolutional neural network (CNN) model with seven layers used in [40] with η ( t ) = 0 . 01 , E = 5 , α = 0 . 00015 , and a batch size of 16. We adopt the ResNet-18 model for CIFAR-100 dataset with η ( t ) = 0 . 01 , E = 7 , α = 0 . 0007 , and a batch size of 64. The learning rate of CIFAR-100 is decayed by 0 . 993 at each communication round.

5.2 Baselines

We compare SpaFL with multiple state of the art baselines that studied sparse model structures in FL. In FedAvg [1], every client trains a global dense model and communicates whole model parameters. FedPM [28] trains and communicates a binary mask while freezing model parameters. In HeteroFL [7], each client trains and communicates p -reduced models, which remove the last 1 -p output channels in each layer. In Fjord [9], each client randomly samples a model from a set of p -reduced models, which drops out p % of filter/neurons in each layer. FedP3 [41] communicates a subset of sparse layers that are pruned by the server for downlink and personalize the remaining layers. Clients only upload the updated remaining layers to the server. FedSpa [4] trains personalized sparse models for clients while maintaining fixed model density during training. Local only performs local training with the introduced pruning method without any communications. For the sparse FL baselines, the average target sparsity is set to 0.5 following the configurations in [28, 7, 9, 41, 4].

5.3 Main Results

In Table 2 and Fig. 2, we present the averaged accuracy, communication costs, number of FLOPs during training, and convergence rate for each algorithm. We consider all uplink and downlink communications to calculate the communication cost of each algorithm. We also provide the details

90

M

70

40

80

60

[%]

70

60

SpaFL

Accuracy [%]

50

SpaFL

Accuracy [%]

30

SpaFL

FedAvg

FedPM

40

FedAvg

FedPM

20

FedAvg

FedPM

40

HeteroFL

30

HeteroFL

HeteroFL

30

Fjord

Fjord

10

Fjord

20

FedP3

20

FedP3

Fedp3

10

FedSpa

10

FesSpa

0

FedSpa

0

100

200

300

400

51

0

100

200

300

400

5

0

200

400

600

800

10001200

1400

Communicatonrounds

Communicatonrounds

Communicatonrounds

of the FLOPs measure in the Supplementary document. We average the model densities of SpaFL when a model achieved the best accuracy during training. From these results, we observe that SpaFL outperforms all baselines while using the least amount of communication costs and number of FLOPs. The achieved model densities are 35.36%, 30.57%, and 35.38%, for FMNIST, CIFAR-10, and CIFAR-100, respectively. We also observe that SpaFL uses less resources and performs better than FedP3, HetroFL and Fjord, which deployed structured sparse models across clients. For FedP3, clients only upload subset of layers, but the server still needs to send the remaining layers. Although FedPM reduced uplink communication costs by communicating only binary masks, its downlink cost is the same as FedAvg. In SpaFL, since the clients and the server only exchange thresholds, we can significantly reduce the communication costs compared to baselines that exchange the subset of model parameters such as HeteroFL and Fjord. Moreover, SpaFL significantly achieved better performance than Local, which did not communicate trained thresholds. Local achieved 51.2%, 50.1%, and 53.6% model densities for each dataset, respectively. We can see that communicating trained thresholds can make models sparser and achieve better performance. This also corroborates the analysis of Theorem 1. Hence, SpaFL can efficiently improve model performance with small computation and communication costs. In Fig. 2, we show the convergence rate of each algorithm. We can see that the accuracy of SpaFL decreases and then keeps increasing. The initial accuracy drop is from pruning while global thresholds are not trained enough. As thresholds keep being trained and communicated, clients learn how to prune their model, thereby gradually improving the performance with less active filter/neurons.

We provide an empirical comparison between SpaFL and the baseline that does not use the update in Section 3.3.3 in Table. 3. We can see that the update (13) can provide a clear improvement compared to the baseline by extracting parameter importance from global thresholds.

Round 40

Round150

Round500

10

10

10

1Filter

20

30

30

30

Conv

40

40

40

50

50

50

60

60

60

20

40

Client

60

80

20

40

Client

60

80

20

40

Client

60

80

(a) Sparsity pattern at round 40

(b) Sparsity pattern at round 150

(c) Sparsity pattern at round 500

In Fig. 3, we show the change of structured sparsity of the first convolutional layer with 64 filters with three input channels on CIFAR-10. We color active filters as black and pruned filters as white. We can see that clients learn common sparse structures across training round. For instance, the 31th and 40th filters are all pruned at round 40. Meanwhile, the 20th filter is recovered at rounds 150 and 500. We can know that SpaFL enables clients to learn optimized sparse model structures by optimizing thresholds. In SpaFL, pruned filter/neurons can be recovered by sharing thresholds. At round 40, filters are pruned with high sparsity. Since premature pruning damages the performance, most filters are recovered at round 150. Then, clients gradually enforce more sparsity to filters along with training rounds as shown in Fig. 3c.

In Tab. 4, we show the performance of SpaFL on a vision transformer using the ViT [42] on CIFAR-10 dataset. We used the same data distribution as done in Tab. 2. We apply our pruning scheme to multiheads attention layers. Since a multiheads attention layer essentially consists of stacked linear layers, we can simply use (1), thereby making sparse attention. We can see that SpaFL can be applied to transformer architectures by achieving the density of around 42% while outperforming FedAvg.

6 Conclusion

In this paper, we have developed a communication-efficient FL framework SpaFL that allows clients to optimize sparse model structures with low computing costs. We have reduced computational overhead by performing structured pruning through trainable thresholds. To optimize the pruning process, we have communicated only thresholds between clients and a server. We have also presented the parameter update method that can extract parameter importance from global thresholds. Furthermore, we have provided theoretical insights on the generalization performance of SpaFL.

Limitations and Broader Impact One limitation of SpaFL is that it cannot explicitly control the sparsity of clients. Since we enforce sparsity through the regularizer term, we need to run multiple experiments to find values for desired sparsity. Another limitation is that our analysis requires a bounded loss function. Meanwhile, in practice, most loss functions may admit bounds that have a large value. For broader impact, SpaFL can reduce not only the computation and communication costs of FL training, but also those of inference phase due to sparsity. Hence, in general, SpaFL can improve the sustainability of FL deployments, and more broadly, of AI.

References

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Sunwoo Lee, Anit Kumar Sahu, Chaoyang He, and Salman Avestimehr. Partial model averaging in federated learning: Performance guarantees and benefits. arXiv e-prints , pages arXiv-2201, 2023.

Pretom Roy Ovi, Emon Dey, Nirmalya Roy, and Aryya Gangopadhyay. Mixed quantization enabled federated learning to tackle gradient inversion attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5045-5053, 2023.

Tiansheng Huang, Shiwei Liu, Li Shen, Fengxiang He, Weiwei Lin, and Dacheng Tao. Achieving personalized federated learning with sparse local models. arXiv preprint arXiv:2201.11380 , 2022.

Xiaofeng Liu, Yinchuan Li, Qing Wang, Xu Zhang, Yunfeng Shao, and Yanhui Geng. Sparse personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems , 2023.

Dongping Liao, Xitong Gao, Yiren Zhao, and Cheng-Zhong Xu. Adaptive channel sparsity for federated learning under system heterogeneity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20432-20441, 2023.

Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. International Conference on Learning Representations , 2021.

Kai Yi, Nidham Gazagnadou, Peter Richtárik, and Lingjuan Lyu. Fedp3: Federated personalized and privacy-friendly network pruning under model heterogeneity. arXiv preprint arXiv:2404.09816 , 2024.

Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems , 34:12876-12889, 2021.

Taehyeon Kim and Se-Young Yun. Supernet training for federated image classification under system heterogeneity. arXiv preprint arXiv:2206.01366 , 2022.

Mi Luo, Fei Chen, Zhenguo Li, and Jiashi Feng. Architecture personalization in resourceconstrained federated learning. In NeurIPS Workshop on New Frontiers in Federated Learning , 2021.

Won Joon Yun, Yunseok Kwak, Hankyul Baek, Soyi Jung, Mingyue Ji, Mehdi Bennis, Jihong Park, and Joongheon Kim. Slimfl: Federated learning with superposition coding over slimmable neural networks. IEEE/ACM Transactions on Networking , 2023.

Zhenheng Tang, Shaohuai Shi, Wei Wang, Bo Li, and Xiaowen Chu. Communication-efficient distributed deep learning: A comprehensive survey. arXiv preprint arXiv:2003.06307 , 2020.

Joost Verbraeken, Matthijs Wolting, Jonathan Katzy, Jeroen Kloppenburg, Tim Verbelen, and Jan S Rellermeyer. A survey on distributed machine learning. Acm computing surveys , 53(2): 1-33, 2020.

Wenke Huang, Mang Ye, Zekun Shi, Guancheng Wan, He Li, Bo Du, and Qiang Yang. Federated learning for generalization, robustness, fairness: A survey and benchmark. arXiv preprint arXiv:2311.06750 , 2023.

Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering , 35(4):3347-3366, 2021.

Sameer Bibikar, Haris Vikalo, Zhangyang Wang, and Xiaohan Chen. Federated dynamic sparse training: Computing less, communicating less, yet learning better. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 6080-6088, 2022.

Yuang Jiang, Shiqiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K Leung, and Leandros Tassiulas. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Ang Li, Jingwei Sun, Binghui Wang, Lin Duan, Sicheng Li, Yiran Chen, and Hai Li. Lotteryfl: empower edge intelligence with personalized and communication-efficient federated learning. In 2021 IEEE/ACM Symposium on Edge Computing (SEC) , pages 68-79. IEEE, 2021.

Xinchi Qiu, Javier Fernandez-Marques, Pedro PB Gusmao, Yan Gao, Titouan Parcollet, and Nicholas Donald Lane. Zerofl: Efficient on-device training for federated learning with local sparsity. arXiv preprint arXiv:2208.02507 , 2022.

Ang Li, Jingwei Sun, Pengcheng Li, Yu Pu, Hai Li, and Yiran Chen. Hermes: an efficient federated learning framework for heterogeneous mobile clients. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking , pages 420-437, 2021.

Vaikkunth Mugunthan, Eric Lin, Vignesh Gokul, Christian Lau, Lalana Kagal, and Steve Pieper. Fedltn: Federated learning for sparse and personalized lottery ticket networks. In Computer Vision-ECCV 2022: 17th European Conference , pages 69-85. Springer, 2022.

Rong Dai, Li Shen, Fengxiang He, Xinmei Tian, and Dacheng Tao. Dispfl: Towards communication-efficient personalized federated learning via decentralized sparse training. In International Conference on Machine Learning , pages 4587-4604. PMLR, 2022.

Sara Babakniya, Souvik Kundu, Saurav Prakash, Yue Niu, and Salman Avestimehr. Revisiting sparsity hunting in federated learning: Why does sparsity consensus matter? Transactions on Machine Learning Research , 2023.

Dimitris Stripelis, Umang Gupta, Greg Ver Steeg, and Jose Luis Ambite. Federated progressive sparsification (purge-merge-tune)+. In Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS , 2022.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 , 2018.

Ang Li, Jingwei Sun, Xiao Zeng, Mi Zhang, Hai Li, and Yiran Chen. Fedmask: Joint computation and communication-efficient personalized federated learning via heterogeneous masking. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems , pages 42-55, 2021.

Berivan Isik, Francesco Pase, Deniz Gunduz, Tsachy Weissman, and Michele Zorzi. Sparse random networks for communication-efficient federated learning. arXiv preprint arXiv:2209.15328 , 2022.

Erum Mushtaq, Chaoyang He, Jie Ding, and Salman Avestimehr. Spider: Searching personalized neural architecture for federated learning. arXiv preprint arXiv:2112.13939 , 2021.

Minh Tri Lê, Pierre Wolinski, and Julyan Arbel. Efficient neural networks for tiny machine learning: A comprehensive review. arXiv preprint arXiv:2311.11883 , 2023.

Junjie Liu, Zhe Xu, Runbin Shi, Ray CC Cheung, and Hayden KH So. Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers. arXiv preprint arXiv:2005.06870 , 2020.

Xiao Zhou, Weizhong Zhang, Zonghao Chen, Shizhe Diao, and Tong Zhang. Efficient neural network training via forward and backward propagation sparsification. Advances in neural information processing systems , 34:15216-15229, 2021.

Amirkeivan Mohtashami, Martin Jaggi, and Sebastian Stich. Masked training of neural networks with partial gradients. In International Conference on Artificial Intelligence and Statistics , pages 5876-5890. PMLR, 2022.

Lu Yin, Gen Li, Meng Fang, Li Shen, Tianjin Huang, Zhangyang Wang, Vlado Menkovski, Xiaolong Ma, Mykola Pechenizkiy, Shiwei Liu, et al. Dynamic sparsity is channel-level sparsity learner. Advances in Neural Information Processing Systems , 36, 2023.

Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Understanding straight-through estimator in training activation quantized neural nets. arXiv preprint arXiv:1903.05662 , 2019.

Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11264-11272, 2019.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 , 2019.

Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. Advances in neural information processing systems , 32, 2019.

Kai Yi, Nidham Gazagnadou, Peter Richtárik, and Lingjuan Lyu. Fedp3: Federated personalized and privacy-friendly network pruning under model heterogeneity. In The Twelfth International Conference on Learning Representations .

Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Fengxiang He, Bohan Wang, and Dacheng Tao. Tighter generalization bounds for iterative differentially private learning algorithms. In Uncertainty in Artificial Intelligence , pages 802812. PMLR, 2021.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189 , 2019.

Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of machine learning research , 18(153):1-43, 2018.

A Experiments

A.1 Implementation Detail

We run all experiments on NVIDIA A100 GPUs with PyTorch. In Table 6, we provide detailed information of model architectures for each dataset. For the FMNIST dataset, we use the Lenet-5Caffe model, which is Caffe variant of Lenet-5. The Lenet model has 430500 of model parameters and 580 of trainable thresholds. For the CIFAR-10 dataset, we use a CNN model of seven layers used in [40]. It has 807366 of model parameters and 1418 of trainable thresholds. The ResNet-18 model is adopted for the CIFAR-100 dataset with 11159232 of model parameters and 4800 of thresholds. We use a stochastic gradient optimizer with momentum of 0.9. For FMNIST with the Lenet model, we use η ( t ) = 0 . 001 , E = 5 , a batch size of 64, and α = 0 . 002 . For CIFAR-10, we use η ( t ) = 0 . 01 , E = 5 , a batch size of 16, and α = 0 . 00015 . For CIFAR-100, we use η ( t ) = 0 . 01 , E = 7 decayed by 0 . 993 at each communication round, a batch size of 64, and α = 0 . 0007 . All trainable thresholds are initialized to zero. We noticed that too large sparsity coefficient α can dominate the training loss, resulting in masking whole parameters in a certain layer. Following the implementation of [31], if a certain layer's density becomes less than 1%, the corresponding trainable thresholds will be reset to zero to avoid masking whole parameters.

For the ViT, we use the patch size of 4, embedding dimension of 128, depth of 6, 8 heads, and set the dimension of linear layers as 256. We use the same setting with the above CIFAR-10 experiments except α = 0 . 0001 and E = 1 .

A.1.1 More details about baselines

We compare SpaFL with sparse baselines that investigated structured sparsity. In FedAvg [1], every client trains a global dense model and communicates whole model parameters. We used the equal weighted average for the model aggregation. FedPM [28] optimizes a binary mask while freezing model parameters. Clients only transmit their arithmetically coded binary masks to the server, and the server broadcasts real-valued probability masks to the clients. We use Adam optimizer with learning rate of 0 . 1 as done in [28]. HeteroFL [7] selects ⌈ pC ⌉ channels of each layer, where 0 ≤≤ 1 and C is the number of channels, to make p reduced submodels. Clients train and communicate p reduced submodels during training. We set p = 0 . 5 following [7]. Fjord [9] samples p from a uniform distribution U ( p min , p max ) . After sampling p , clients train p reduced submodel by selecting the first ⌈ pC ⌉ channels of each layer. We set p min = 0 . 4 and p max = 0 . 6 [9]. We provide the learning rates of the baselines in the following table. FedP3 [41] communicates a subset of sparse layers that are pruned by the server for downlink and personalize the remaining layers. Clients only upload the updated remaining layers to the server. We choose 'OPU2' method, which uniformly selects two layers for clients from the entire network. Hence, clients only upload these chosen layers to the server. For the pruning methods, we adopted the ordered dropout for structured sparsity. FedSpa [4] trains personalized sparse models for clients while maintaining fixed model density during training. The initial pruning rate is set to be 0.5 and decayed using cosine annealing.

A.2 Proof of Theorem 1

We next present the detailed proof of Theorem 1. The proof is inspired by [23] and [43] To facilitate the proof, we first provide the definition of differential privacy and key lemmas from [43].

Definition 1. (Differential privacy). A hypothesis A is ( ϵ, δ ) - differentially private for any hypothesis subset A 0 and adjacent datasets S and S ′ which differ by only one example such that

Lemma 1. (Theorem 4 in [43]) For an iterative algorithm A i at round i , define the update rule as follows:

If for any fixed A i -1 , M i is ( ϵ i , δ ) private, then {A i } T i =0 is ( ϵ ′ , δ ′ ) differntially private such that ϵ ′ = √ 2 ∑ T i =0 ϵ 2 i log 1 ˜ δ + ∑ T i =0 ϵ i exp( ϵ i ) -1 exp( ϵ i )+1 ,

Lemma 2. (Theorem 1 in [43]) For an ( ϵ, δ ) private hypothesis A , the training dataset size D ≤ 2 ϵ 2 ln 16 exp( -ϵ ) δ , and the loss function ||L|| ∞ < 1 , we have

Proof. The overall proof follows [23] by showing that SpaFL is an iterative machine learning algorithm that satisfies differential privacy at each round. Then, we can use lemmas from [43] that provide generalization bound to differential private algorithm. One major difference from [23] is that we have global thresholds not global parameters.

We first define notations for the proof. The diameter of the gradient space is defined as M g = max w,z,z ′ , τ ||∇ F ( w, τ ; z ) -∇ F ( w, τ ; z ′ ) || , where z is an input-output pair. We also denote G k, B = 1 |B| ∑ z ∈B h k ( ˜ w k ; z ) as the average of h k ( ˜ w k ) over B . We use P as probability distribution and P A as the probability distribution conditioned on A .

From Algorithm 1, it is clear that SpaFL is iteratively optimizing global thresholds τ in each client at every round. We now derive the differential privacy of (9) in Algorithm 1. Here, each client calculates h k using its subset of local data. As done in [23], we assume that additive Gaussian noise sample is added in (9) in Algorithm 1 for the analysis. Since we always have global thresholds at round t , (9) can be seen as sampling a mini-batch I ( t ) from D = ∪ k D with mini-batch size ξ and we let

B ( t ) = S I ( t ) . Then, for fixed τ ( t -1) and two adjacent sample sets S and S ′ , we have

where τ ( t ) = τ ( t -1) -η ( t -1) ( G S I ( t -1) + N (0 , σ 2 I ) ) and G S I ( t -1) = 1 N ∑ N k =1 G k,S I k ( t -1) . We define η ( t -1) τ ′ = τ ( t -1) -τ ( t ) -η ( t -1) G S I ( t -1) , then we can rewrite (21) as below

Since τ ∼ τ ( t -1) -η ( t -1)( G S I ( t -1) + N (0 , σ 2 I )) due to added Gaussian noise samples, τ ′ ∼ N (0 , σ 2 I ) . Then, following the definition of differential privacy, we define

where (23) is from the definition of Gaussian distribution. We now denote G S I ( t -1) -G S ′ I ( t -1) in (24) as v . We derive the bound of || v || as follows

where (25) is from the definition of G and (26) is from the definition of the diameter of gradient M g . Note that some elements of h k ( ˜ w k ; z ) will be zero since we do not calculate gradients of pruned filter/neurons due to structured sparsity. Hence, we multiply the current model density to derive (26).

We next bound ⟨ τ ′ , v ⟩ in (24). Since ⟨ τ ′ , v ⟩ ∼ N (0 , || v || 2 σ 2 ) , we have the following inequality using Chernoff Bound as

We define δ as follows

Then, with the probability of 1 -δ with respect to τ ′ , we can derive the bound of (24) as follows

Following Lemma 1 and (13) in [23], we can derive that each round in Algorithm 1 is (˜ ϵ, ξ D δ ) differntially private, where ˜ ϵ is given as

where ξ is the size of S I ( t -1) . Subsequently, we apply Lemma 1 to have ( ϵ ′ , δ ′ ) differential privacy for T communication rounds. Lastly, we finish the proof by using Lemma 2.

A.3 Convergence Rate Analysis

We derive the convergence rate of SpaFL. Since we only communicate thresholds τ , we derive the convergence rate of the global thresholds. In SpaFL, we simultaneously update τ and w , and it is analytically challenging to track the update of τ for multiple local epochs E . As such, we analyze the convergence of SpaFL under the special case with E = 1 . We leave a more general convergence analysis with multiple local epochs for future works. We now make two assumptions [44] as follows

Assumption 1. (smoothness) F k ( · ) is M -smooth for τ and client k , ∀ k

Assumption 2. (Unbiased stochastic gradient) The stochastic gradient h k is an unbiased estimator of the gradient ∇ τ F k , respectively, for client k, ∀ k , such that

Then, we have the following convergence rate

Theorem 2. For γ ( t ) = η ( t )(1 -α (1 -Mη ( t )) 2 ) and the largest number of parameters connected to a neuron or filter n max in > 0 in a given model, we have

From (9), thresholds τ ( t ) are updated using parameter gradients g k ( t ) , k ∈ S . We can expect that the thresholds will converge when parameters w k , ∀ k , converge. We can see that the sparsity regularizer coefficient α impacts convergence. As α increases, we can quickly enforce more sparsity to the model. However, a very large α will damage the performance as γ ( t ) decreases in (1). We can also observed that the convergence depends on the difference between the received global thresholds τ ( t ) and the updated thresholds τ k ( t ) . Hence, a very large change to the global thresholds will lead to a significantly different binary mask in the next round. Then, local training can be unstable as parameters have to adapt to the new mask. Therefore, from Theorem 2, we can capture the tradeoff between the computing cost and the learning performance in terms of α .

Proof. We first consider the case in which global thresholds converge. We have the following update rule for global thresholds as

We take the expectation over the randomness in client scheduling and stochastic gradients as follows

Hence, clearly τ will eventually converge if 1 N E || ∑ N k =1 ∇ τ F k ( ˜ w k ( t ) , τ ( t )) || 2 converges. We next show that this conditional statement holds in our SpaFL framework.

From the M -smoothness of the loss function in Assumption 1, we have

To facilitate the analysis, we first derive τ k ( t ) -τ ( t ) as below

Then, we can change (36) as follows

We next take the expectation to the above inequality and use Assumption 2 as below

We first bound A using ⟨ a, b ⟩ ≤ || a || 2 + || b || 2 2 as below

We now further bound B as

where n max in is the largest number of parameters connected to a neuron or filter in a given model, ( a ) is from | w | ≤ 1 in Section 3.2.1, and the last inequality is from the M -smoothness of F k . By combining A and B with taking expectation, we have

By arranging the above inequality, we have

where γ ( t ) = η ( t )(1 -α (1 -Mη ( t )) 2 ) . We now further bound ( A ) in (43). From Assumption 1, we have the following

Based on (44), we can bound (43) as below

From (45), we can bound the averaged aggregated gradients with respect to thresholds as below

By summing the above inequality from t = 0 to T -1 , we can obtain the result of Theorem 2.

Based on Theorem 2, we can derive the convergence rate with the Big-O notation by following the steps in [8]. We first assume η = 1 / √ T and 0 ≤ α < 1 . Then, we can bound 1 γ ( t ) ≤ 2 η ( t )(1 -α ) . By

replacing η ( t ) and γ ( t ) with their assumed value and bound into the above convergence rate, we have the following bound

where A = ∑ T -1 t =0 ∑ N k =1 E ||∇ τ F k ( ˜ w k ( t ) ,τ ( t )) -∇ τ k F k ( ˜ w k ( t ) ,τ ( t )) || 2 MN , B = ∑ T -1 t =0 4 α || exp( -τ ( t )) || 2 , C = M 2 n max in G 2 / 2 , and D = ∑ T -1 t =0 ∑ N k =1 || τ ( t ) -τ k ( t ) || 2 N .

A.4 Communication Costs Measure

We calculate the communication cost of SpaFL considering both uplink and downlink communications. At each round t , sampled clients transmit their updated thresholds to the server. Hence, the uplink communication costs can be given by

where τ num is the number of thresholds of a given model. In downlink, the server broadcasts the updated global threshold to sampled clients. Hence, the downlink communication costs can be given as below

Therefore, total communication costs can be given by T × ( CommUp + Commdown ) .

A.5 FLOPs Measure

We calculate the number of FLOPs during training using the framework introduced in [32]. We consider a convolutional layer with an input tensor X ∈ R N × C × X × Y , parameter tensor W ∈ R F × C × R × S , and output tensor O ∈ R N × F × H × W . Here, the input tensor X consists of N number of samples, each of which has X × Y dimension. The parameter tensor W has F filters of C channels with kernel size R × S . The output tensor O will have F output channels with dimension H × W for N samples. During forward propagation, a filter in W performs convolution operation with the input tensor X to produce a single value in the output tensor O . Hence, we can approximate the number of FLOPs as N × ( C × R × S ) × F × H × W . Since we use a sparse model during forward propagation, the number of FLOPs can be reduced to ρ × N × ( C × R × S ) × F × H × W , where ρ = || p || 0 || W || 0 is the density of the parameter matrix W . For the backpropagation, we calculate it as 2 times of that of forward propagation following [45].

For a fully connected layer with input tensor X ∈ R N × X and parameter tensor W ∈ R X × Y , the input tensor X is multiplied with W during the forward propagation. Hence, with the density of W , we can calculate the number of FLOPs for the forward propagation as ρ × N × X × Y . In backpropagation, we follow the same process for convolutional layers.

We also consider the number of FLOPs to perform line 6 in Algorithm 1 for updating the local models from global thresholds. Sampled clients first have to decide update directions by doing summation of connected parameters at each neuron/filter (sum operation). Then, they update their local models using the received global thresholds (sum and multiply operations). This corresponds to 1 . 5 × d FLOPs, where d is the number of model parameters. Then, the total number of FLOPs during one local epoch at round t can be approximately given by

B Change of sparsity patterns on CIFAR-10

Here, we present the change of sparsity patterns of different layers on CIFAR-10.

Round 40

Round 150

Round 500

10

10

10

Conv 2 Filter

20

Conv 2 Filter

20

Filter

20

30

30

N

Conv

30

40

40

40

50

50

50

60

09

60

0

20

40

60

80

Q

20

40

60

80

0

20

40

60

80

Client

Client

Client

Round40

Round150

Round500

20

20

20

1Filter

dense 1 Filter

40

40

40

60

60

60

dense

80

80

装

80

100

100

100

120

120

120

Q

20

40

Client

60

80

0

20

40

Client

60

80

20

40

Client

60

80

Dir = 0.1

Dir =0.5

Dir=1.0

10

10

Conv1Filter

20

20

30

30

Conv

40

40

40

50

50

50

60

60

60

0

20

40

09

08

0

20

40

60

80

0

20

40

60

80

Client

Client

Client

B.1 Change of Model Sparsity patterns in conv2

B.2 Change of Model Sparsity patterns in dense1

From Figs. 4 and 5, we can observe that clients learn common sparsity patterns across layers by communicating thresholds.

B.3 Change of Model Sparsity patterns with different data heterogeneity

From Fig. 6, we can see that as the data heterogenity decreases, clients share more similar sparsity patterns across their filters.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿✴✴♥✐♣/a115✳❝❝✴ ♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿ ✴✴♥✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ❤/a116/a116♣/a115✿✴✴♥❡✉/a114✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❊/a116❤✐❝/a115●✉✐❞❡❧✐♥❡/a115 ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes] NA

Justification: We do not include pretraiend language models or image generators.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ♣❛♣❡/a114/a115✇✐/a116❤❝♦❞❡✳❝♦♠✴❞❛/a116❛/a115❡/a116/a115 has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: we do not have crowdsourcing experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not have human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The authors consider distributed training of sparse models, by optimizing over the thresholds used to prune the models.

STRENGTHS:
I am not an expert of deep learning but the results look convincing enough.

WEAKNESSES:
I would be interested in a discussion and comparison with other approaches that train sparse models, such as
* Meinhardt et al. "Prune at the Clients, Not the Server: Accelerated Sparse Training in Federated Learning," preprint arXiv:2405.20623, 2024.  
* Yi et al. "FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models," preprint arXiv:2403.09904, 2024. 

and references therein.

The experiments should investigate more the influence of the level of heterogeneity, because this is what makes it difficult to identify the sparsity pattern.

QUESTIONS:
No question

LIMITATIONS:
The technical limitations are discussed at the end of the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-13

comments: SUMMARY:
This paper introduces SpaFL, a federated learning framework that enhances communication efficiency and minimizes computational overhead by optimizing sparse model structures. They achieve this goal by defining a trainable threshold which leads to structured sparsity. Since the server and clients only exchange thresholds, SpaFL is able to reduce communication overhead. Furthermore, these trainable thresholds also lead to low computing costs.

STRENGTHS:
1. The paper is generally well-written and easy to follow

2. The idea of using trainable thresholds is new and effective for achieving both communication efficiency and low computational overhead.

3. They perform valid experiments with the baseline methods and some image datasets. The accuracy performance of their method is better than others.

WEAKNESSES:
1. I recommend the authors to do additional experiments with NLP tasks.

2. In Figure 8 of the FedPM paper, the accuracy achieved by FedPM is higher than what is reported in this paper. Could you clarify the reason for this discrepancy? Is it due to differences in experimental setup and hyperparameters? If so, could you ensure that all conditions are consistent for the CIFAR-100 experiments and compare your method to the original accuracy reported in the FedPM paper?

QUESTIONS:
1. If you apply this method to the transformer based models, how can you define trainable thresholds? To be specific, you gave an explanation for a convolutional layer in section 3.1, and can we just similarly define for self attention modules?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper suggests communicating the threshold instead of the model parameters in federated learning. Through empirical validations on popular benchmarks, the proposed method, SpaFL, is shown to have lower computational overhead and achieve relatively good results.\n\nSTRENGTHS:\nCommunicating the threshold instead of the model weights seems to be interesting. Preliminary empirical results on popular benchmarks seem to have shown that this idea works with lower communication overhead and relatively good performance.\n\nWEAKNESSES:\n1. The main figure, Figure 1, is not informative and very confusing. I would expect to see a high-level explanation of how the threshold for each client is selected, whether they are the same for each layer, and how the selected thresholds are combined. At this moment, I can only see that all local models have the same threshold and they are directly combined at the server. Due to the misalignment between the main figure and the illustration in section 3.1, the necessary details to fully understand the main idea of this paper are not easy to follow. The writing in this section could also be improved.\n\n2. In Theorem 1, the authors provided a generalization bound. However, the intuition and analysis for this bound are missing. At this moment, I do not have a good sense of, in a practical or special case, how large the right-hand side probability of Equation 14 is? How large must the minimal training data size $D$ be? Is it practical? Also, this bound only measures the distance between the empirical risk and the expected risk; how can this be regarded as a generalization bound? To my understanding, a generalization bound should be between the training set and the testing set. Furthermore, could you please provide the convergence rate and communication complexity of the proposed method? These are very important for the theoretical analysis of FL.\n\n3. The baselines are outdated, such as Fjord (NeurIPS21) and HeteroFL (ICLR21). Can you please compare more recent methods, which can be easily found, such as [FedPAC](https://arxiv.org/abs/2306.11867) and [FedCR](https://proceedings.mlr.press/v202/zhang23w.html)? I would expect to see more comparisons with newer methods. They do not need to be the ones mentioned above, but I believe it is critical to determine whether the proposed method is still interesting to explore at this moment.\n\nQUESTIONS:\nI would expect a README file to better understand your code efficiently.\n\nLIMITATIONS:\nThe major limitations include the unclear presentation of the proposed method, lack of analysis of the generalization bound, absence of convergence and communication efficiency analysis, and the use of outdated empirical baselines.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}