## 3 Dual Sparsity Pruning Federated Learning Framework
### Overview of DualPFL  
DualPFL is designed to address the challenges of the huge communication overhead in heterogeneous client data distributions and varying computational capabilities of federated learning scenarios. As depicted in Figure 1, the workflow of DualPFL can be divided into the following five steps:  
1. For the initial global round, the server randomly selects a set of devices and distributes the initialized model to them. \begin{table}
\begin{tabular}{c c} \hline \hline
**Symbol** & **Descriptions** \\ \hline \(N\) & Total number of clients \\ \(T\) & Total number of global epochs \\ \(t\) & The index of the \(t\)-th global epoch \\ \(L\) & Number of local epochs for clients in a round \\ \(l\) & The index of the \(l\)-th local epoch \\ \(D\) & Total dataset for all clients \\ \(g\) & Gradient of the model \\ \(\mathbf{\theta}_{i}\) & Local model parameters of the \(i\)-th client \\ \(\mathbf{W}_{i}\) & Weight matrices of the \(i\)-th client \\ \(\Delta\mathbf{\theta}_{i}\) & Model update of the \(i\)-th client \\ \(\mathbf{M}\) & The sparse matrix \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Descriptions of main notations. 2. The device calculates the inverse of Kullback-Leibler (KL) divergence to obtain the feature \(p\), performs dataset-aware sparse pre-training, and uploads the pruned models to the server. 3. The server then aggregates the model using a scale-based approach, evaluates the performance of the global model, and assesses the sparsity ratio. 4. Starting from the second round, the device performs local training via stochastic gradient descent (SGD) and stores the model updates. 5. The accumulated model gradient updates and is then subjected to an element-wise multiplication (Hadamard product) with the dynamically adjusted sparse matrix to derive the channel-pruned updates. Steps 3-5 are repeated until the model converges or the maximum number of rounds are reached. As summarized in Algorithm 1, We first propose a dataset-aware approach that incorporates the reciprocal of regularized Jensen-Shannon (JS) divergence as a feature into the initial stage of sparse pruning for each client. Specifically, we first perform a non-linear mapping based on the JS divergence between the local dataset distribution and the standard distribution. The reciprocal of this divergence is used as a feature value, denoted as \(p_{i}\), for the \(i\)-th client. Then, we introduce a mask matrix \(\mathbf{M}_{i}\) with dimensions consistent with the model parameters to achieve sparse pruning. The pruning is guided by the learned importance of filters, ensuring that the local model is adaptive to the local data distribution. Each element in this mask is generated from a _Bernoulli_ (\(p_{i}\)) distribution. We apply this binary mask to the weight tensor to conduct sparse pre-training. For each channel filter, this mechanism results in a probability \(p_{i}\) of retaining the filter unit and a probability of \(1-p_{i}\) of setting it to zero. Figure 1: Our proposed DualPFL framework. To maintain channel sparsity during global aggregation, we incorporate an L1-norm regularization term into the global loss function. After the first global round involving pre-training with the sparse mask, the server distributes the trained model \(\theta\) to the edge devices. Subsequently, each edge device trains the model via a local dataset. During this stage, there is no need for further sparsification of the gradients. Instead, each client starts to execute \(L\) steps of SGD, which is formulated as follows:  
\[\theta_{i,t}^{l}=\theta_{i,t}^{l-1}-\eta g_{i,t}^{l}. \tag{2}\]  
```
1:local models, local datasets, etc.