# Advanced Financial Fraud Malware Detection Method
## 4 Experimental Results and Discussion
### Selected Model and Decision Threshold  
Table 14 presents a comparison of the evaluation results for each model. The main goal of this study is to accurately detect and classify benign and malware apps to ensure the effectiveness of the model in real-world operational environments. Upon evaluating various classifiers, LightGBM was selected as the most suitable model owing to its exceptional performance across key metrics. LightGBM achieved the highest accuracy (0.999996) and best F1-Score (0.968564), demonstrating an excellent balance between precision and recall, which is essential for handling imbalanced datasets. It also recorded the highest AUPRC  
Figure 19: ROC–AUC curve obtained using CatBoost.  
Figure 18: Confusion matrix of CatBoost classifier with two target classes.  
(0.987537), highlighting its superior ability to differentiate between benign and malicious apps even in challenging scenarios. Compared with other models, such as random forest and CatBoost, LightGBM consistently balanced the precision, recall, and F1-Score more effectively. Logistic Regression, although computationally simple, struggles significantly with imbalanced data, making it impractical for this task. LightGBM was chosen for its outstanding detection performance and computational efficiency, making it ideal for deployment in resource-constrained or real-time environments. Its scalability and robust generalization across various evaluation metrics further solidified its position as the best choice, outperforming traditional models and advanced gradient-boosting models, such as XGBoost and CatBoost. In summary, LightGBM proved to be the most practical and effective solution for this application.  
Additional experiments were conducted to further enhance the performance of LightGBM and determine the optimal decision threshold. The experimental results are presented in Table 15. Based on these results, we selected 0.5 as the optimal decision threshold value. A fixed decision threshold (0.5) was ultimately selected for its ability to balance precision and recall, simplicity of implementation, and suitability for operational environments. Although alternative thresholds, such as 0.603 or dynamic values, offer potential benefits for specific use cases, their added complexity or limited incremental improvements make them less practical for general applications. The final results of our proposed malware detection model exhibited an accuracy of 99.9958%, a precision of 96.6507%, a recall of 97.4277%, and an F1-Score of 97.0376.  
* Threshold 1: Threshold that always changes according to the model and date. For each test date, D, the model was fitted from D-70 to D-15. Subsequently, the threshold  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Model** & **Accuracy** & **Precision** & **Recall** & **F1-Score** & **AUROC** & **AUPRC** \\ \hline Logistic Regression & 0.849444 & 0.000376 & 0.924700 & 0.000751 & 0.953914 & 0.014239 \\ \hline Random Forest & 0.999994 & 0.987150 & 0.906815 & 0.945278 & 0.999461 & 0.982923 \\ \hline LightGBM & 0.999996 & 0.964526 & 0.972635 & 0.968564 & 0.999559 & 0.987537 \\ \hline XGBoost & 0.999992 & 0.919447 & 0.951350 & 0.935127 & 0.999145 & 0.963108 \\ \hline CatBoost & 0.999995 & 0.929774 & 0.987480 & 0.957759 & 0.999988 & 0.977388 \\ \hline \hline \end{tabular}
\end{table}
Table 14: Overall comparison of the tested classifiers with two target classes.  
Figure 20: Precision-Recall curve obtained using CatBoost.  
that produced a high F1-Score for the benign and malware samples from D-14 to D-1 was selected.
* Threshold 2: The default threshold value was fixed at 0.5.
* Threshold 3: The threshold was fixed at a single value of 0.603, which produces high F1-Scores for all 92 datasets measured ex post facto (for reference).