## 6 Conclusions  
This paper proposes a novel framework for malware author group classification based on human-intervenable classification and graph embedding. We extracted features from malware using both static and dynamic analyses. With the selectively extracted features, we built a malware-feature bipartite graph and performed a KNN classification model based on graph embedding. We also removed the less significant features of efficiency. To make the framework human-intervenable, we suggested intra- and inter-class closeness \begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \begin{tabular}{c} **k-NN** \\ **(w/o Give Up)** \\ \end{tabular} & \begin{tabular}{c} **Intra-Class** \\ **(\(\theta=1.5\))** \\ \end{tabular} & \begin{tabular}{c} **Expert 1** \\ \end{tabular} & \
\begin{tabular}{c} **Expert 2** \\ \end{tabular} &
\begin{tabular}{c} **Average** \\ \end{tabular} \\ \hline \# classified & 1941 & 1811 & 1860 & 1884 & 1872 \\ \# correct & 1819 & 1742 & 1777 & 1789 & 1783 \\ \(F1_{\mu}\) & 93.7\% & 96.2\% & 95.5\% & 95.0\% & 95.25\% \\ \# give up & 0 & 130 & 81 & 57 & 69 \\ \hline \# correct & - & - & 1858 & 1846 & 1852 \\ after manual & & & (1777 + 81) & (1789 + 57) & (1783 + 69) \\ inspection & & & & & \\ \hline \(F1_{\mu}\) after & - & - & 95.7\% & 95.1\% & 95.4\% \\ manual & & & & & \\ inspection & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Effectiveness of man-machine collaboration. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \begin{tabular}{c} **k-NN** \\ **(w/o Give Up)** \\ \end{tabular} & \begin{tabular}{c} **Inter-Class** \\ **(\(\theta=3\))** \\ \end{tabular} & \begin{tabular}{c} **Intra-Class** \\ **(\(\theta=2\))** \\ \end{tabular} & \
\begin{tabular}{c} **Intra-Class** \\ **(\(\theta=1.5\))** \\ \end{tabular} &
\begin{tabular}{c} **Intra-Class** \\ **(\(\theta=1.2\))** \\ \end{tabular} \\ \hline \# classified & 1941 & 1670 & 1715 & 1763 & **1811** & 1856 \\ \# correct & 1819 & 1613 & 1657 & 1701 & **1742** & 1774 \\  & & & (1613 + 44) & (1613 + 88) & **(1613 + 129)** & (1613 + 161) \\ \hline \(F1_{M}\) & 85.1\% & 90.0\% & 89.8\% & 89.5\% & **88.9\%** & 87.8\% \\ \(F1_{\mu}\) & 93.7\% & 96.6\% & 96.6\% & 96.5\% & **96.2\%** & 95.6\% \\ \(F1_{w}\) & 93.7\% & 96.7\% & 96.7\% & 96.5\% & **96.2\%** & 95.6\% \\ \hline \# give up & 0 & 271 & 226 & 178 & **130** & 85 \\ \hline \% of & 100\% & 86.0\% & 88.4\% & 90.8\% & **93.3\%** & 95.6\% \\ classified & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Effectiveness of inter-close and intra-close closeness. The accuracy and postponing rate are balanced when \(\theta=1.5\). as metrics for determining the ambiguity of classification. We also developed visualized box plot results for human experts. We conducted experiments using a real-world malware dataset labeled by cybersecurity experts. The results confirmed the effectiveness of intra- and inter-class closeness as metrics and the effectiveness and efficiency of the human-intervenable framework. In our future work, we plan to additionally use the control flow and the call graph information as our malware features. Since the control flow and the call graph information have been recognized as unique characteristics of a program, we believe incorporating such graph information into our malware-feature bipartite graph will be a promising research direction to more accurate author group classification on malware. We are also planning to try various existing visualization techniques for malware. Based on an excellent survey [32], there have been malware visualization methods which seem to be applicable to our work (e.g., 2D/3D displays or geometrically-transformed displays). We plan to apply them to the graphical user interface for helping our human experts determine the author group of a target malware. We will also try to evaluate which visualization makes our human experts classify given malware the most accurately. Conceptualization, S.-J.P. and E.K. ; methodology, D.-K.C. and E.K. ; software, J.H. ; validation, D.-K.C.