## 4 Experiments and Discussion
### Ablation Studies
#### 4.3.1 Ablation Study on Local Attention Mechanism  
To verify the effectiveness of our local attention mechanism, we simply use the networks shown in Figure 3 for convenience. We use the network in Figure 3a as our baseline. The only difference between network (a) and (b) is that (a) uses the original Transformer encoder, but (b) employs our well-designed dual attention Transformer encoder (DAT encoder). The networks take the API sequence as input. Features are extracted via six encoder layers, like the Transformer, and then sent into two linear layers and one softmax activation, outputting the final probabilities. Moreover, to find the best value of \(D\) in the local attention module proposed in Section 3.1, we set \(D\) ranging from 1 to 6 and compare the accuracy results to discuss. \begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Methods** & **Features** & **Samples** & **Families** & **Accuracy** \\ \hline Malware Image + GIST [44] & File content & 63,002 & 531 & 0.7280 \\ Malware Image + CNN [45] & File content & 10,868 & 9 & 0.9176 \\ Malware Image + GRU-SVM [46] & File content & 9339 & 25 & 0.8492 \\ BBIS + CARL [47] & API calls & 3131 & 28 & 0.8840 (F1) \\ NLP(TF-IDF) + SVM [14] & API calls & 23,080 & 10 & 0.8654 \\ Category Vector + CNN [16] & API calls & 23,080 & 10 & 0.8797 \\ Frequency Vector + RF [23] & API calls & 23,080 & 10 & 0.8005 \\ Embedding + RNN [20] & API calls & 23,080 & 10 & 0.8690 \\ Encoder (Embedding + CNN) + Bi\_LSTM [21] & API calls & 23,080 & 10 & 0.9021 \\ Random Transformer Forest [48] & API calls & 23,080 & 10 & 0.9330 \\ Ours & API calls & 23,080 & 10 & 0.9606 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison with previous methods on Dataset One. \begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Methods** & **Features** & **Samples** & **Families** & **Accuracy** \\ \hline Category Vector + CNN [16] & API calls & 33,240 & 16 & 0.7782 \\ Frequency Vector + RF [23] & API calls & 33,240 & 16 & 0.6685 \\ Embedding + RNN [20] & API calls & 33,240 & 16 & 0.8690 \\ Voting Experts + Confidence Weighted [49] & API calls, Actions & 33,240 & 16 & 0.8570 \\ Encoder (Embedding + CNN) + Bi\_LSTM [21] & API calls & 33,240 & 16 & 0.8051 \\ Random Transformer Forest [48] & API calls & 33,240 & 16 & 0.8703 \\ Ours & API calls & 33,240 & 10 & 0.8859 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Comparison with previous methods on Dataset Two. We compare the effect and best setting of our local attention module in Table 6. Our baseline is the Transformer encoder performance, which using the Transformer encoder (global attention) resulted in an accuracy of 0.7719 \(\pm\) 0.0049. The introduction of the DAT (local attention) encoder led to notable improvements in accuracy, where all results are above 0.80. Specifically, the best performance was achieved when D = 2, with an accuracy of 0.8368 \(\pm\) 0.0038. This represents a substantial gain of 6.4% over the Transformer encoder. Moreover, the DAT encoder is more robust than the Transformer encoder, and the results show less deviation compared to 0.0049. And we can find that although the best setting is D = 2, even when D was increased, the accuracy remained competitive, demonstrating the effectiveness of the local attention mechanism. The experimental results strongly suggest that incorporating a local attention module (DAT encoder) is beneficial for improving performance on the given task. The optimal configuration, in this case, was found to be D = 2, and this setting was chosen for subsequent experiments. Based on the DAT encoder (D = 2), we test against 10 kinds of malware. As can be seen in Table 7, the F1-scores for different malware categories varied, indicating the model's ef