# Similarity-Based Malware Classification Using Graph Neural Networks
## 4 Performance Analysis
### Performance Comparison  
The ssdeep and TLSH techniques are based on fuzzy hash algorithms, and they are widely used to identify malware. Consequently, fuzzy hash algorithms are compared and used to generate a baseline. If the similarity score output by ssdeep is greater than zero, then the malware pair are in the same family; else, they are in different families. The threshold of the TLSH is set to 100, as recommended in Reference [39]. The malware pair are in different families when the distance score of the TLSH exceeds 100; otherwise, the pair are in the same family. Although ssdeep and TLSH are mainly used for comparing the similarity, they can also perform the classification function by transforming samples into related hash values and mapping to related categories. First, the sample is transformed into hash values. Then, the hash values are mapped with the training dataset. When the sample is similar to a \begin{table}
\begin{tabular}{c c c c c} \hline
**Accuracy** & **Precision** & **Recall** & **F1-Score** & **AUC** \\ \hline
0.920 & 0.929 & 0.910 & 0.919 & 0.920 \\ \hline \end{tabular}
\end{table} 
Table 4: Performance of similarity model. Figure 11: Confusion matrix and ROC curve of similarity model. specific category in the training dataset, the input sample can be classified as that category. Therefore, Table 5 compares the performance of the classification model with ssdeep and TLSH. From the experimental result, the method proposed by this study is higher than traditional fuzzy hash methods. This is because fuzzy hash methods transform whole files into hashes, while the graph classification method caches the relationship between the executed functions. Reference [25] proposed the MSM_GCN strategy, based on a study of similarity matching of malware families based on a graph similarity metric using graph convolutional networks. However, they did not provide sufficient experimental details concerning, for example, data pre-processing and model hyper-parameters. Therefore, the MSM_GCN that is used for comparison is consistent with the original concept but may have been differently implemented. Herein, a fixed feature value of one is added to each node and the graph feature is obtained from global_max_pool. Table 6 compares the performances of the similarity model herein with the ssdeep, TLSH, and MSM_GCN techniques. With respect to the similarity among malware families, finding variants that belong to the same family is costlier than finding those that belong to different families. Hence, the precision is more important than the recall in this study. The widely used fuzzy hash method takes the raw malware sample as input, which does not contain the relation of function call graph and therefore has lower precision compared with graph methods. The precision of the proposed method is 0.196 higher than that of MSM_GCN technique, and the proposed method performs best among the four compared methods. The families in the test dataset are considered during model training. Therefore, an unseen dataset is used herein to test the effectiveness of family verification; it contains three new families and 500 samples that did not appear in the training dataset \(d\). The function call relationship and the function assembly content are obtained by analyzing  
Figure 12: Visualization of test dataset using PCA. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Technique** & \
\begin{tabular}{c} **Test Dataset** \\ **(Accuracy)** \\ \end{tabular} &
\begin{tabular}{c} **Test Dataset** \\ **(Precision)** \\ \end{tabular} \\ \hline Ssdeep [6] & 0.627 & 0.572 \\ TLSH [7] & 0.634 & 0.593 \\
**Proposed method** & **0.934** & **0.938** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Classification model performance comparison with previous works. the malware to generate a graph that represents its functional structure during the model training period. The proposed model's overall performance decreases by about 0.2 when tested on the unseen dataset, similar to the MSM_GCN technique.