# Advanced Financial Fraud Malware Detection Method
## 4 Experimental Results and Discussion
### Comparative Analysis and Statistical Validation of Proposed Model  
To validate the performance of our proposed financial fraud malware detection model, we conducted a comprehensive statistical comparison using five feature set variations (Variation1-Variation5) and the proposed model (Table 16). The analysis was performed across four key performance metrics: accuracy, recall, precision, and F1-Score. This section presents the statistical analysis and visualizations using a one-way Analysis of Variance (ANOVA) and box plots to demonstrate the superiority of our proposed model.  
To assess the statistical significance of the performance differences across the six feature sets, we applied one-way ANOVA for each performance metric. The null hypothesis (H\({}_{0}\)) assumes that the mean performance is the same across all feature sets, while the alternative hypothesis (H\({}_{1}\)) assumes that at least one feature set performs significantly better than the others [64]. The ANOVA results were as follows:  
* Accuracy: F-Statistic = 56.22, \(p\)-value = 3.91 \(\times\) 10\({}^{-47}\)
* Recall: F-Statistic = 58.19, \(p\)-value = 1.60 \(\times\) 10\({}^{-48}\)
* Precision: F-Statistic = 25.25, \(p\)-value = 6.16 \(\times\) 10\({}^{-23}\)
* F1-Score: F-Statistic = 43.92, \(p\)-value = 4.42 \(\times\) 10\({}^{-38}\)  
Figures 21 and 22 present histograms of F-statistics and \(p\)-values for different metrics. As the \(p\)-values for all the metrics were significantly smaller than 0.05, we reject the null hypothesis. This confirms that statistically significant differences existed in the performance of the feature sets for all metrics. Table 17 and Figure 23 present the box plots generated to provide a visual representation of the variability and central tendencies of each feature set across the four performance metrics. The box plots illustrate that the proposed model consistently achieves higher median values and less variability than the other variations, indicating a more stable and superior performance across all datasets [71].  
\begin{table}
\begin{tabular}{c c} \hline \multicolumn{2}{c}{**Feature Sets**} \\ \hline Variation1 & App information, App permissions, App service \\ \hline Variation2 & App information, App permissions, App service, User activity \\ \hline Variation3 & App information, App permissions, App service, User information \\ \hline Variation4 & App information, App permissions, App service, App package name statistics \\ \hline Variation5 & App information, App permissions, App service, User activity, App package name statistics \\ \hline Our Proposed & App information, App permissions, App service, User activity, User information, App package name statistics \\ \hline \end{tabular}
\end{table}
Table 16: Variations of feature sets for feature selection techniques.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline LightGBM + Threshold 1 & 0.999996 & 0.964526 & 0.972635 & 0.968564 \\ \hline LightGBM + Threshold 2 & 0.999958 & 0.966507 & 0.974277 & 0.970376 \\ \hline LightGBM + Threshold 3 & 0.999959 & 0.969151 & 0.972267 & 0.970706 \\ \hline \hline \end{tabular}
\end{table}
Table 15: Selection of decision threshold.  
Figure 23: Boxplots for evaluation metrics for feature selection techniques.  
Figure 21: Histogram of f-statistics for different metrics.  
Figure 22: Histogram of \(p\)-values for different metrics.  
The ANOVA results, supported by visual insights from the box plots, demonstrate that the proposed model significantly outperforms the five variations across all performance metrics. The inclusion of additional features, such as user activity, user information, and app package name statistics, in the proposed model led to notable improvements in accuracy, recall, precision, and F1-Score. These enhancements render the proposed model a reliable solution for detecting fraudulent malware in real-world applications.