## 4 Experiments and Discussion
### Findings and Limitations  
We compare and discuss through lots of experiments, including (1) the performance comparison between the method proposed in this paper and various types of machine learning and deep learning models; (2) the role of the local attention mechanism; and (3) the role of a two-stage training strategy. By comparing the experimental results, we can see the following:  
(1) Our proposed method can achieve an accuracy of 0.9606, but most classical methods cannot exceed 0.90. In addition, this paper is validated on two datasets, and although the performance of all the models on the second dataset is generally degraded, the \begin{table}
\begin{tabular}{l c c} \hline \hline
**Category** & **F1-Score** & **Overall Accuracy** \\ \hline Worm & 0.95 & \\ P2P-Worm & 0.62 & \\ Trojan-Spy & 0.93 & \\ Net-Worm & 0.95 & \\ Packed & 0.99 & \\ Trojan-Downloader & 0.90 & 0.96 \\ Trojan-PSW & 0.95 & \\ Trojan-Ransom & 0.91 & \\ Misc & 0.83 & \\ Trojan-FakeAV & 0.99 & \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Quantitative results of two-step inference. Figure 7: The confusion matrix (with data on accuracy) of the results of two-step reference. From top to bottom, from left to right, there are Worm, P2P-Worm, Trojan-Spy, Net-Worm, Packed, Trojan-Downloader, Trojan-PSW, Trojan-Ransom, Misc, and Trojan-FakeAV. method in this paper still outperforms the others, which suggests that the method in this paper is very effective and capable of achieving the latest SOTA results;
2. We compare two works based on the Transformer architecture. Overall, the performances of the three methods are nice, which illustrates the effect of natural language modeling, especially the latest Transformer architecture, on the processing capability of API sequential calls. We further designed the fusion of local attention mechanisms based on the local feature relationships between API calls, which can enhance the feature-capturing ability even further. And this can also be compared more carefully in ablation experiments. By exploring the DAT encoder, we can find that the local attention module can indeed achieve better results than the original Transformer encoder;
3. Our ablation analysis for Step 1 also further verifies that the introduction of the Siamese Network can improve the performance to a certain extent, with an improvement in accuracy from 0.837 to 0.87. In the second step, we use the Normal Vector to simplify the process of malware classification and obtain a more generalizable category representation vector that can further improve the accuracy of the model, with an increase in the average accuracy from 0.87 to 0.92. This suggests that, compared to randomly selected samples, the Normal Vector obtained in the second training step is better at capturing malware category features;
4. Finally, we made a further attempt to adapt to the hierarchical relationship between the samples by performing a two-step inference, trying to classify the parent category first and then the subcategories. Experiments show that this strategy is effective and can improve the accuracy from 0.92 to 0.96, which utilizes the richer data in the parent category to improve the stability of the Normal Vectors of the subcategories with limited data and to alleviate the problem of data scarcity for specific malware types (e.g., P2P worms). However, from the analysis of our results, there are still some limitations of underperformance; accordingly, we point out the limitations and future directions of the work in this paper. 1. The challenge of data scarcity: Despite the efforts to address the data scarcity issue through the training steps, in multiple comparison experiments, relatively high variance in F1-scores can be found across different categories, e.g., P2P Worms and Misc, which still exhibit low F1-scores. The performance of the model is limited by the availability of representative data, and the strategy to deal with rare data needs to be further explored;
2.