# Malware Detection Using Memory Analysis Data in Big Data Environment
## 5 Experiments and Evaluation
### Results and Comparison  
In the study, the performance of the models established with machine learning and deep learning algorithms in binary classification was evaluated. Accuracy, Precision, Recall, F1-score, Accuracy, and ROC-AUC parameters were calculated to compare the performances. The values of performance metrics for the nine machine learning and deep learning methods used are shown in Table 5.  
According to Table 5, the algorithm that gives the best accuracy rate is Logistic Regression with 99.97%. The Gradient Boosted Tree algorithm follows a Logistic Regression with 99.94%. Looking at the comparison chart in Figure 9, it is seen that all algorithms achieve high accuracy rates in the classification of the malware dataset. The algorithm with the lowest accuracy rate is the Multilayer Perceptron algorithm with 97.67%. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Base Classifier** & **Precision** & **Recall** & **F1-Score** & **Accuracy** & **ROC-AUC** \\ \hline Decision Tree & 99.73\% & 99.85\% & 99.79\% & 99.79\% & 99.98\% \\ Random Forest & 99.97\% & 99.84\% & 99.90\% & 99.90\% & 99.76\% \\ Naive Bayes & 97.02\% & 99.85\% & 98.42\% & 98.41\% & 84.25\% \\ Logistic Regression & 99.98\% & 99.97\% & 99.97\% & 99.97\% & 100.00\% \\ Gradient Boosted Tree & 99.97\% & 99.91\% & 99.94\% & 99.94\% & 99.98\% \\ Linear SVC & 98.69\% & 99.57\% & 99.14\% & 99.14\% & 99.76\% \\ MLP & 96.42\% & 98.98\% & 97.68\% & 97.67\% & 99.23\% \\ DNN & 99.46\% & 99.64\% & 99.55\% & 99.56\% & 99.56\% \\ LSTM & 99.17\% & 99.69\% & 99.43\% & 99.43\% & 99.43\% \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Precision, Recall, F1-Score, Accuracy, and ROC-AUC results. When we compare the AUC values of the models to evaluate their ability to distinguish classes, it is seen that the Naive Bayes algorithm has a rate of 84.25%, which is considerably lower than other machine learning and deep learning algorithms. Looking at the ROC curve given in Figure 9(a), it is seen that the models other than Naive Bayes have quite high AUC values. The algorithm with the highest AUC value is Logistic Regression with 100%. The Gradient Boosted Tree algorithm has a 99.98% AUC value. Figure 9(b) shows the PRC graph of the models. It is seen that as the Recall values of the Naive Bayes algorithm increase, the Precision value decreases rapidly after a point and falls below the 0.5 Precision threshold. When the Recall values of the LibSVM and MLP algorithms decrease, it is seen that there is some fluctuation in the Precision values. However, in general, it can be said that the algorithms have a perfect balance. According to the Confusion matrices given in Figure 10, the Logistic Regression algorithm misclassified only 2 benign and 3 malwares. In other words, the Logistic Regression algorithm performed the most successful classification with the least number of misclassifications. Decision Tree and Gradient Boosted Tree algorithms both classified 3 benign as malware. However, Gradient Boosted Tree classified less malware as benign. In this way, it can be said that the Gradient Boosted Tree performs a more successful classification than the Decision Tree and Random Forest. In addition, Decision Tree and Naive Bayes algorithms  
Figure 9: Performance Comparison of ML and DL Models. Figure 10: (**a**) ROC-AUC: comparison of ML and DL Models; (**b**) PRC-AUC: comparison of ML and DL Models. also seem to misclassify the same number of malwares. However, Decision Tree has made few misclassifications in benign classification and is more successful than Naive Bayes. MLP algorithm misclassified 320 benign and 89 malwares, making it the most misclassified algorithm. Naive Bayes and Linear SVC algorithms drew attention by misclassifying a high number of benign. When examined in more detail, the Naive Bayes algorithm's FN rate is lower than FP. In other words, it can be said that the reason for the low success rate of the Naive Bayes algorithm is its failure in classifying the benign ones.