# A Federated Incremental Learning Algorithm Based on Dual Attention Mechanism Mechanism
## 3 Federated Incremental Learning Algorithm
### Dual Attention Mechanism Module  
The original design concept of federated learning aims to jointly train high-quality global models on the client side without revealing privacy. For example, mobile shopping malls, banks, and other apps with high-quality customer information can recommend suitable shopping and financial products through joint model training without revealing user information. Many large enterprises have trained high-performance global models to help small and medium-sized enterprises in their later development. However, due to the imbalance of samples between clients, there will be a problem that the noise generated by clients with a large sample size when participating in the training will also significantly impact the results of the final trained global model. To this end, given the problem of large samples and considerable noise, consider adding a channel attention mechanism on the client-side, perform feature compression in the spatial dimension to obtain a feature map with a global receptive field, and then learn through fully connected layers (FC) relationship between channels. Finally, multiplying the learned weight coefficients of each channel with all the elements of the corresponding channel can help to minimize the impact of noise on the final model training results while obtaining the characteristics of the respective overall samples of each client. At the same time, federated learning joins incremental learning [48, 49] to perform dynamic task training. Learning at the same simultaneouslyi-catmtttttion increments, will lead to the imbalance of sample importance and the problem that the importance of features is difficult to capture. For this reason, it is difficult to capture the imbalance of features. We consider adding a feature attention mechanism [50, 51] when the global model is aggregated, to enhance the capture performance of the model training for critical feature information. Therefore, we consider adding a dual attention mechanism module that simultaneously channels attention and features attention to the federated learning framework. The dual attention mechanism module is shown in Figure 3.  
Figure 2: Federated incremental learning framework. In Figure 3, the first dual attention mechanism module is to add a channel attention mechanism to the client to help obtain the characteristics of the overall samples of each client while minimizing the impact of noise on the final model training results. There are many models of the local model of the client, such as LSTM, CNN, and other algorithms combined with the SE channel attention module. The architecture of CNN combined with channel attention is shown in Figure 4. The specific process of the client-side neural network model is shown in Figure 5.  
As shown in Figure 4, the SE module is added after the first convolutional layer of the local model, because the number of channels in the layers too far behind is too large, which is easy to cause overfitting. If the feature map is too small, we use Improper operation will introduce a large proportion of non-pixel information. More importantly, close to the classification layer, the effect of attention is more sensitive to the classification results, and it is easy to affect the decision of the classification layer. Therefore, after placing it in the first convolutional layer, each layer of the convolutional network has 16 convolution  
Figure 4: Neural network model. Figure 5: Neural network model flow chart. Figure 3: Dual attention mechanism module. kernels, and each convolution kernel corresponds to a feature channel. Channel attention can allocate resources between each convolution channel. The final output is obtained by learning the degree of dependence of each channel and adjusting different feature maps according to the degree of dependence. Therefore, we can add the channel attention mechanism to help focus on the overall characteristics of the sample and reduce the impact of noise on the final model training results.