# MLMD--A Malware-Detecting Antivirus Tool Based on the XGBoost Machine Learning Algorithm
## 4 Evaluation
### Comparison of MLMD and VirusTotal
#### 4.2.2 Comparison of Classifications  
Table 17 and the charts shown in Figure 11 show the percentage distribution of classifications made by our program and VirusTotal on malicious samples.  
As is evident, our _MLMD_ program classified 94.29% of all samples as malicious, which was a correct classification. The remaining 5.71% were samples classified as suspicious, which can also be considered as a partially correct answer, considering the malicious samples.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Class** & **Count** \\ \hline Malicious & 70 \\ Benign & 35 \\ Total & 105 \\ \hline \hline \end{tabular}
\end{table}
Table 16: Test sample composition.  
As far as VirusTotal is concerned, a total of 71.17% of systems considered the samples to be malicious. This value represents the sensitivity of VirusTotal as a whole service, not individual scanners. However, a number of systems incorrectly flagged the samples to be benign, in 26.83% of all cases.  
Table 18 and the charts shown in Figure 12 show the percentage distribution of classifications made by our own _MLMD_ and VirusTotal on benign samples.  
As far as the benign samples are concerned, the only correct classification of the sample is its classification as benign. Our _MLMD_ program classified 82.86% of all samples as benign, which was a correct classification. It made an incorrect classification for 17.14% of the samples (it classified the samples to be suspicious); thus, one of our classification models made an incorrect classification. However, none of the samples were classified to be malicious by the program.  
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **MLMD** & **VirusTotal** \\ \hline All samples & 70 & 70 \\ Malicious samples (\%) & 94.29 & 71.17 \\ Suspicious samples (\%) & 5.71 & - \\ Benign samples (\%) & 0 & 26.83 \\ \hline \hline \end{tabular}
\end{table}
Table 17: Comparison of classifications of malicious samples.  
Figure 11: Comparison of predictions of malicious samples.  
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **MLMD** & **VirusTotal** \\ \hline All samples & 35 & 35 \\ Malicious samples (\%) & 0 & 1.94 \\ Suspicious samples (\%) & 17.14 & - \\ Benign samples (\%) & 82.86 & 98.06 \\ \hline \hline \end{tabular}
\end{table}
Table 18: Comparison of classifications of benign samples.  
Figure 12: Comparison of classifications of malicious samples.  
The VirusTotal system--to be precise, the systems it uses--performed better on benign samples. In the case of benign samples, 98.06% of all systems correctly labelled the samples as benign and only 1.94% incorrectly as malicious.  
Table 19 shows the number of samples incorrectly classified as malicious. We defined the success threshold to be 35, which is approximately half of VirusTotal's available detection systems. Thus, this is the number of samples that were flagged correctly as malicious by fewer than 35 systems and flagged incorrectly as benign by most systems. We found eight such samples. However, our program classified only two of these samples incorrectly (i.e., as suspicious but not malicious).  
A similar comparison for benign samples can be seen in Table 20. As in the previous case, we have defined the success threshold to be 35. Thus, the table shows the number of samples that were flagged correctly as benign by fewer than 35 systems and flagged incorrectly as malicious by most systems. However, considering VirusTotal, there were no such samples; the majority of samples was correctly detected to be benign by most systems. Our program flagged six samples as suspicious incorrectly.  
In all of the above comparisons, we saw a good success rate of the _XGBoost_ algorithm, especially in classifying the selected group of malicious samples, which confirms the success observed on the test data. However, it is important to mention that all decision-tree-based algorithms proved to be the best in detecting malware, outperforming other algorithms by a wide margin.