# A Malware Detection Framework Based on Semantic Information of Behavioral Features
## 4 Dataset and Evaluation Methods
### Experimental Results
#### 4.3.2 Ablation Studies  
The proposed model is made up of several components that can be flexibly tuned, such as feature processing, CNNs, and BiGRU. To explore the effects of different configurations, we conduct several groups of comparative experiments by fixing other structures and  
\begin{table}
\begin{tabular}{l c c c c} \hline  & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline NB & 0.8198 & 0.8518 & 0.7768 & 0.8126 \\ KNN & 0.9175 & 0.9556 & 0.8767 & 0.9145 \\ DT & 0.9178 & 0.9184 & 0.9180 & 0.9182 \\ Text-CNN & 0.9443 & 0.9587 & 0.9298 & 0.9440 \\ BiLSTM & 0.9743 & 0.9766 & 0.9723 & 0.9758 \\ CNNs-BiGRU & 0.9828 & 0.9870 & 0.9789 & 0.9827 \\ \hline \end{tabular}
\end{table}
Table 4: Comparison with baseline models.  
Figure 8: Comparisons of ROC curves of different models.  
changing only the test components. The results of these experiments are the foundation for determining the structure of the final model.
* **Feature processing:** a set of experiments is used to verify the detection effectiveness of various feature processing methods, networks only the contextual semantic information about the sequence of API calls, networks only the semantic structure for the names of API calls and statistical features, and networks with both.
* **CNNs:** by a set of experiments to observe the model effects of API sequences of different lengths, two convolutional layers with the first filter stride of 3 and the second filter stride of 5; three convolutional layers with kernel strides of 3, 4, and 5; three convolutional layers with kernel strides of 3, 5, and 7; and four convolutional layers with kernel strides of 3, 4, 5, and 6.
* **BiGRU:** a set of experiments are to understand the impact of the pact of the BiGRU layer, the model has no BiGRU, only one unidirectional GRU, and BiGRU.  
As shown in Figure 9, the absence of any part causes the model's performance to degrade, indicating that each feature we deal with has a will impact on our test model. The model that combines both features achieves the highest performance, especially with respect to accuracy and F1-Score, significantly better than the other two sets of experiments. It is significant to note that the module lacking information regarding the semantics of the API sequence obtained the worst performance, indicating the importance of semantic information about API call context.  
For the application of the number of convolutional layers, Figure 10 illustrates the performance for each configuration. The convergence is slower and the final performance is poorer when the convolutional layer size is 2. The model performance is higher when the number of convolutional layers is 3 than when it is 2. Among them, the patterns with filter sizes of 3, 4, and 5 have higher performance than those with filter sizes of 3, 5, and 7. In addition, the performance does not bring improvement as the convolution layer count grows from 3 to 4. Therefore, we chose a three-layer convolutional layer structure in our model with applied filter sizes of 3, 4, and 5.  
Figure 9: Comparison analysis of accuracy, precision, recall, and F1-Score of processing methods.  
Figure 11 illustrates a comparison of various numbers of GRUs. It is clear that BiGRU model has significantly better accuracy, recall, and F1-Score than the other two models, which indicates the importance of BiGRU for this detection model. It is noticeable that as the quantity of GRUs grows, the convergence of the model gradually accelerates and the performance improves, reflecting that learning the contextual relationship of API calls helps improve the model's performance.  
Figure 11: Comparative analysis of accuracy, precision, recall, and F1-Score of BiGRU model.  
Figure 10: Comparison analysis of accuracy, precision, recall, and F1-Score of the number of convolutional layers.