# A Novel Malware Detection Model in the Software Supply Chain Based on LSTM and SVMs
## 5 Experimental Results and Analysis
### Experimental Results and Analysis
#### 5.4.2 Evaluation of LSTM-BO-SVM Model Efficiency
As shown in Figure 9, on the ClaMP dataset, the LSTM-BO-SVM model demonstrates superior precision at 98.7% for malware, ranging from 0.2% to 13.5% higher than the other models. For benign software, this model achieves 97.7% precision, which is 0.7% lower than the BiLSTM-BO-SVM model but still higher than the other models, ranging from 0.1% to 15.8%. As shown in Figure 10, on the CICMalDroid-2020 dataset, the LSTM-BO-SVM model demonstrates superior precision at 98.6% for malware, ranging from 0.6% to 15.2% higher than other models. For benign software, the precision is also 98.6%, which is lower than the LSTM-SVM and BiLSTM-BO-SVM models by 1.1% and 0.1%, respectively, but higher than the other models, which range from 0.1% to 16.8%. The experimental findings demonstrate the high precision with which the LSTM-BO-SVM model can classify both malware and benign software. The BO-SVM model achieved good detection results on benign software, but its detection results on malware were worse than benign software. The precision of the LSTM model and the LSTM-SVM model varies greatly between benign and malware. In particular, on the ClaMP dataset, the LSTM-SVM model classifies malware with significantly higher precision than benign software, while on the CICMalDroid-2020 dataset, the situation is reversed. The LSTM model performs similarly to the LSTM-SVM model. In the experiments, the precision of the LSTM-SVM model obtained from each experiment fluctuated greatly, which may be partially attributed to the default hyperparameter settings of the SVM model. The RNN-BO-SVM model and the BiLSTM-BO-SVM model achieved better detection results, and the BiLSTM-BO-SVM model was better than the RNN-BO-SVM model, but still not as good as the LSTM-BO-SVM model. 4. Recall Rate Comparison  
Figures 11 and 12 exhibit the recall rate comparisons of the above six models on the ClaMP and the CICMalDroid-2020 datasets, respectively. As shown in Figure 11, on the ClaMP dataset, the LSTM-BO-SVM model has a recall rate of 97.9% for malware, which is 0.2% and 0.6% lower than the BO-SVM and BiLSTM-BO-SVM models, respectively, but still higher than the other models, with a range from 1.3% to 19.3%. The LSTM-BO-SVM model's recall rate for benign software is 98.6%, which is 1% less than the LSTM-SVM model's recall but greater than the other models' recall rates, which range from 1.2% to 15.8%. Figure 11: Recall rate comparisons of six models on ClaMP. As shown in Figure 12, on the CICMalDroid-2020 dataset, the LSTM-BO-SVM model's recall rate for malware is 96.9%, which is 0.3% lower than the BiLSTM-BO-SVM model and 0.3% to 0.9% higher than the other models. For benign software, the recall rate of the LSTM-BO-SVM model is 99.4%, which is 0.3% to 8.5% higher than the other models. According to the experimental findings, the LSTM-BO-SVM model demonstrates a high recall rate on both datasets, whether detecting benign software or malware. The BiLSTM-BO-SVM model comes second, while the LSTM-SVM model performs relatively mediocrely in terms of recall. The excellent recall performance of the LSTM-BO-SVM model is attributed to two key factors: the optimization of hyperparameters and the LSTM structure's ability to efficiently process sequence data. However, the poor performance of the LSTM-SVM model may be due to the lack of effective hyperparameter optimization. Although the BiLSTM-BO-SVM model performs better, its performance is slightly lower than that of the LSTM-BO-SVM model, possibly due to the complexity of the model structure. 5. F1-Score Comparison  
Figures 13 and 14 exhibit the F1-Score comparisons of the above six models on the ClaMP and the CICMalDroid-2020 datasets, respectively. Figure 12: Recall rate comparisons of six models on CICMalDroid-2020. Figure 13: F1-Score comparisons of six models on ClaMP. As shown in Figure 13, on the ClaMP dataset, the LSTM-BO-SVM model achieves an F1-score of 98.3% for malware, which is higher than the other models, ranging from 0.2% to 10.4%.