# A Federated Incremental Learning Algorithm Based on Dual Attention Mechanism Mechanism
## 3 Federated Incremental Learning Algorithm
### Dual Attention Mechanism Module
This mechanism can automatically consider the weight of the relationship between the server model and the client. In iterative training, continuously updating the parameters reduces the weighted distance between the server and the client model, and the expected distance between the server and the client model is minimized. The optimization objective is calculated as shown in Formula (7). \[J(\omega_{t}^{s},\omega_{t+1}^{1},...,\omega_{t+1}^{K})=arg\min_{\omega_{t+1} ^{s}}\sum_{k=1}^{n}[\frac{1}{2}att_{k}D(\omega_{t}^{s},\omega_{t+1}^{k})^{2}] \tag{7}\]  
Among them, \(\omega_{t}^{s}\) is the model parameter of the server in the \(t\)th communication, and \(\omega_{t+1}^{K}\) is the model parameter of the client k in the \((t+1)\)th communication. \(D(\cdot,\cdot)\) is the distance between the two sets of neural parameters calculated using the Euclidean distance formula, \(att_{k}\) is the important weight of the client model. The hierarchical soft attention method is used to capture the hierarchical importance of the neural network in multiple local models, and it is aggregated into the global model as feature attention to achieve the optimal server and client. The distance between the models is minimized, the expected function of the previous formula is derived to obtain the gradient, and K clients perform gradient descent to update the parameters of the global model. Figure 2:0  
Figure 2:1  
The importance weight of the client model \(att_{k}\) is calculated by hierarchical soft attention. The server-side model is used as a query value, and the client-side model is used as a key value to calculate the attention score of each layer in the neural network. The attention formula is shown in Equation (10). It should be noted that due to the  
Figure 2:2 Feature attention mechanism based global aggregation. incremental learning, the number of neurons output by the last layer of the model fully connected is the number of dataset classes, which changes dynamically, resulting in loading the local model parameters at this time to the server. When calculating the attention score of each layer together with the global model in the last communication, there will be a problem of weight mismatch. Therefore, before calculating the attention score of each layer, we need to average the weights of the last fully connected layer in all client models, and then assign them to the previous layer parameters of the global model of the latest communication. Figure 2:3  
Figure 2:4  
where \(K\) is the number of clients, \(\omega^{l}\) is the model parameter of the \(l\)th layer of the server, \(\omega_{k}^{l}\) is the model parameter of the \(l\)th layer of the \(k\)th local client, \(l\in[1,L]\). We take the p-norm of the difference between the matrices as the similarity value of the query and key values of the \(l\)th layer and use the softmax function for the similarity value to obtain the attention value of the \(l\)th layer of client \(k\), and the feature attention of the entire client \(att_{k}=\{att_{k}^{1},att_{k}^{2},...,att_{k}^{L}\}\).