# A Novel Malware Detection Model in the Software Supply Chain Based on LSTM and SVMs
## 5 Experimental Results and Analysis
### Experimental Results and Analysis
#### 5.4.2 Evaluation of LSTM-BO-SVM Model Efficiency  
In order to evaluate the performance of the LSTM-BO-SVM model in malware detection in the SCC, the following models were selected in this paper for comparative analysis:  
1. RNN-BO-SVM model: First, use an RNN for pre-classification, and then use the BO algorithm to optimize the SVM model; finally, use the optimized SVM model to complete the classification. 2. BiLSTM-BO-SVM model: First, use Bidirectional Long Short-Term Memory (BiLSTM) for pre-classification, and then use the BO algorithm to optimize the SVM model; finally, use the optimized SVM model to complete the final classification. 3. BO-SVM model: First, use the BO to optimize the SVM model, and then use the optimized SVM model to complete the final classification. 4. LSTM-SVM model: First, use the LSTM model for pre-classification, and then use the SVM model to complete the classification. 5. Basic LSTM model: Use the LSTM model alone to complete the final classification. In this paper, to ensure the consistency and comparability of the experiments, the same values were used for the same parameters in each model. In order to present the experimental results more comprehensively, this paper used the evaluation metrics introduced in Section 5.3 to evaluate the six models mentioned above. In addition, in order to visualize the detection effect of the LSTM-BO-SVM model and to effectively reveal the correct and incorrect predictions of the model in each category, a heat map of the confusion matrix of the LSTM-BO-SVM model was drawn. 1. Confusion matrix The confusion matrices are shown in Figures 5 and 6 for the ClaMP dataset and the CICMalDroid-2020 dataset, respectively. It is evident from Figures 5 and 6 that the vast majority of samples are concentrated on the main diagonal of the confusion matrix. This phenomenon indicates that the LSTM-BO-SVM model can achieve a high degree of accuracy and discrimination when classifying samples. The sample points on the main diagonal represent correctly classified instances, while those on the off-diagonal represent misclassified instances. Therefore, a high proportion of samples in the main diagonal is intuitive evidence of the excellent detection performance of the model. The high accuracy of the LSTM-BO-SVM model on the two datasets further confirms its effectiveness in the SSC malware detection task. Overall, the findings of the confusion matrix demonstrate that the LSTM-BO-SVM model achieves a good level of detection effectiveness. Figure 5: Confusion matrix for ClaMP. this perspective, this shows that the LSTM-BO-SVM model possesses a better detection ability on this dataset. Regarding the CICMalDroid-2020 dataset, the LSTM-BO-SVM model also shows its superiority. The accuracy of the suggested model is greater than those five models, ranging from 2.2% to 14.7%. The lower accuracy of the LSTM model may be attributed to its difficulty in adequately extracting features when the amount of data are insufficient. In addition, the LSTM-SVM model lacks an optimization algorithm to determine the hyperparameters of the SVM classifier, which may lead to its unsatisfactory detection effect. Although the RNN-BO-SVM model uses the BO algorithm for optimization, its accuracy is still lower than that of the LSTM-BO-SVM model. This may be due to the vanishing gradient problem faced by RNN when dealing with long-term dependencies, which affects its ability to capture long-distance dependencies in the data. In contrast, there is little difference in accuracy between the BiLSTM-BO-SVM model and the LSTM-BO-SVM model, which indicates that the bidirectional LSTM structure alleviates the gradient vanishing problem of RNN to some extent, thus improving the performance of the model. 3. Precision Comparison  
Figures 9 and 10 show the precision comparison of the six models on the ClaMP and CICMalDroid-2020 datasets, respectively.