# Efficient and Secure Federated Learning for Financial Applications
## 3 Efficient and Secure Federated Learning Based on Gradient and Mask Sparsification
### Time-Varying Hierarchical Gradient Sparsification  
The hierarchical gradient sparsification of the network can reduce the loss caused by the sparsification process. In the early stages of training, network parameters usually undergo significant changes. However, after a certain number of update iterations, the amplitude of the changes decreases. Capitalizing on this feature, we propose a time-varying hierarchical gradient sparsification (THGS) algorithm. This algorithm builds upon hierarchical sparsification and reduces the sparsity rate with an increase in iteration rounds. Eventually, it reaches a set lower bound, as illustrated in Algorithm 1.  
Each participant in THGS calculates a model update, denoted as \(g\), based on its local private data. Assuming that the model update of the \(i\)-th layer is \(g_{i}\), THGS first identifies the Top-\(k\) largest values of \(g_{i}\). At the same time, THGS sets a threshold \(\delta\) to the \(k\)-th largest value in \(g_{i}\). Then, THGS calculates a mask vector \(\widetilde{g_{i}}\) for sparsification. Specifically, we set the element to 1 in \(\widetilde{g_{i}}\) if the value on the corresponding index of \(g_{i}\) is greater than the threshold \(\delta\). Conversely, if the corresponding value of \(g_{i}\) is smaller than the threshold \(\delta\), the element is set to 0 in \(\widetilde{g_{i}}\). Next, THGS computes a sparse model update \(g_{i-sparse}\) using the Hadamard product of \(g_{i}\) and \(\widetilde{g_{i}}\), namely, \(g_{i-sparse}=g_{i}\bigodot\widetilde{g_{i}}\). Simultaneously, THGS needs to calculate a residual model update \(g_{i-residual}\) of \(g_{i}\), which will be kept locally and accumulated in the subsequent communication rounds. Specifically, \(g_{i-residual}=g_{i}-g_{i-sparse}\). Eventually, each participant uploads the sparse model update \(g_{i-sparse}\) to the server for further aggregation, which significantly reduces the upload communication overhead.  
We assume that the initial sparsity rate is \(s_{0}\), \(\alpha\) is a constant attenuation factor, and the lower limit of the sparsity rate is \(s_{min}\). In a deep neural network, the sparsity rate \(s_{i}\) of the \(i\)-th layer parameter is:  
\[s_{i}=\begin{cases}s_{0},&i=1\\ s_{i-1}\cdot\alpha,&if\ \ s_{i-1}\cdot\alpha>s_{min}\\ s_{min},&else\end{cases} \tag{1}\]The process of sparsification can enhance the security of gradient updates in Federated Learning (FL). In FL, gradient attacks can be conducted by the server, which takes the simulated sample data as the input and the gradient sent by the client as the label to infer the true input data. This can be achieved by taking the difference between the updated gradient after the simulated sample data are sent to the initial model for training and the updated gradient actually sent by the client as the penalty function. The simulated sample data are then continuously optimized until the gradient calculated by the simulated sample data and the gradient sent by the client reach a close degree of similarity. By doing this, the simulated sample data will be similar to the local sample data of the client and the original value of the client's sample will be obtained.  
The server can only upload a small percentage of the real gradient data such as one percent or one thousandth, which weakens the server's ability to carry out a gradient attack. The label itself, as a penalty function, is incomplete data, making the simulated sample fitted by this method far from the real sample. Sparsification can further reduce the amount of data uploaded to the server, which ultimately weakens the server's ability to carry out a successful gradient attack.