# Malware Classification Based on Shallow Neural Network
## 4 Experiments and Evaluations
### Experimental Results and Analysis  
#### 4.2.1 Model Performance  
Tables 4 and 5 respectively show the classification performance of the proposed SNNMAC model for two datasets when the n of the n-gram was three. In this experiment, we randomly selected 3000 samples as the training set and 6000 samples as the testing set. In the second data set, one tenth of the data set was randomly selected from each family as the testing set, and the n value of three was also selected for the experiment. For each family, there were only two results: belonging or not belonging to this family. The standard f1 value was used here.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Family Name** & **Precision** & **Recall** & **F1 Value** \\ \hline Ramnit & 0.9677 & 0.9901 & 0.9788 \\ \hline Lollipop & 0.9992 & 0.9976 & 0.9984 \\ \hline Kelihos\_ver3 & 1 & 1 & 1 \\ \hline Vundo & 1 & 1 & 1 \\ \hline Simda & 1 & 0.9091 & 0.9524 \\ \hline Tracur & 0.9863 & 0.9796 & 0.9829 \\ \hline Kelihos\_ver1 & 0.9974 & 0.9871 & 0.9922 \\ \hline Obfuscator.ACY & 0.9825 & 0.9615 & 0.9719 \\ \hline Gatak & 0.9951 & 0.9980 & 0.9965 \\ \hline weighted average & 0.9921 & 0.9918 & 0.9920 \\ \hline \hline \end{tabular}
\end{table}
Table 4: The performance of the proposed model with 3-g.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Number** & **Malware Family Name** & **Sample Size** \\ \hline
1 & Autoit & 723 \\
2 & Delf & 1847 \\
3 & Hotbar & 1937 \\
4 & Onlinegames & 2543 \\
5 & Sytro & 1288 \\
6 & Fosniw & 411 \\
7 & Renos & 593 \\
8 & Installcore & 713 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The malware sample dataset (2).  
Figures 6 and 7 show visual representations of the classification result confusion matrix of two datasets. The confusion matrix is a special matrix used to show the result of multi-classification tasks, with each column representing the prediction class and each row representing the actual class. The matrix is normalized due to the big difference in the number of family samples.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Family Name** & **Precision** & **Recall** & **F1 Value** \\ \hline Autoit & 0.9924 & 1 & 0.9962 \\ \hline Delf & 0.9343 & 0.9925 & 0.9625 \\ \hline Hotbar & 1 & 1 & 1 \\ \hline Onlinegames & 0.9961 & 0.9454 & 0.9701 \\ \hline Sytro & 1 & 0.9956 & 0.9978 \\ \hline Fosniw & 0.9841 & 1 & 0.9920 \\ \hline Renos & 0.9905 & 0.9830 & 0.9867 \\ \hline Installcore & 0.9969 & 0.9969 & 0.9969 \\ \hline weighted average & 0.9844 & 0.9836 & 0.9837 \\ \hline \hline \end{tabular}
\end{table}
Table 5: The performance of the proposed model with 3-g.  
Figure 6: The normalized confusion matrix of the classification result (1).  
The experimental results show that the SNNMAC algorithm obtained a good classification effect for most malware families. It achieved 100% classification precision on five families. For the first dataset, except for the Simba family with a small number of samples, it achieved an f1 value of 0.95 due to the low recall rate. All other malware families obtained an f1 value greater than 0.97, a classification precision above 96.7% and a recall rate more than 96.1%. For the second dataset, except for the Delf family attaining a precision value of 0.93 and the Onlinegames family attaining a recall value of 0.95, all other malware families obtained f1 values greater than 0.97, classification precisions above 98.4% and recall rates of more than 98.3%.