## 5 Evaluation
### Experimental Results
#### 5.2.2 Effectiveness of Intra-Class Closeness  
Next, we evaluated the effectiveness of the intra-class closeness based on the 271 inter-class outliers determined in the previous experiment. We selected each outlier and examined whether \(\frac{d_{t,1}}{d_{t,2}}>\theta\): if so, our KNN classifier classifies this malware; otherwise, it \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & & **Total** & **\# Inliers** & **\# Outliers** \\ \hline \multirow{2}{*}{Total} & \# malwares & 1941 & 1670 & 271 \\  & \# correct/incorrect & 1819(93.7\%)/122 & 1613(96.6\%)/57 & 206(76.0\%)/70 \\ \hline \multirow{2}{*}{Group A} & \# malwares & 1220 & 1074 & 146 \\  & \# correct/incorrect & 1176(96.4\%)/44 & 1074(98.2\%)/19 & 121(82.9\%)/25 \\ \hline \multirow{2}{*}{Group B} & \# malwares & 85 & 71 & 14 \\  & \# correct/incorrect & 63(74.1\%)/22 & 61(85.9\%)/10 & 2(14.3\%)/12 \\ \hline \multirow{2}{*}{Group C} & \# malwares & 85 & 69 & 16 \\  & \# correct/incorrect & 70(82.4\%)/15 & 61(88.4\%)/8 & 9(56.3\%)/7 \\ \hline \multirow{2}{*}{Group D} & \# malwares & 72 & 65 & 7 \\  & \# correct/incorrect & 58(80.6\%)/14 & 57(87.7\%)/8 & 1(14.3\%)/6 \\ \hline \multirow{2}{*}{Group E} & \# malwares & 479 & 391 & 88 \\  & \# correct/incorrect & 452(94.4\%)/27 & 379(96.9\%)/12 & 73(14.3\%)/21 \\ \hline \multirow{2}{*}{} & \(F1_{M}\)**(total)** & 85.1\% & 90.0\% & 51.1\% \\ \hline \multirow{2}{*}{} & \(F1_{\mu}\)**(total)** & 85.1\% & 96.6\% & 76.0\% \\ \hline \multirow{2}{*}{} & \(F1_{w}\)**(total)** & 85.1\% & 96.7\% & 74.8\% \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Effectiveness of inter-class closeness. postpones the classification and transfers the malware to human experts. The experiment was conducted under various values for \(\theta\). In Table 2, \"Y/correct\" indicates the number of malware samples satisfying the intra-class closeness, and the number of correctly classified samples. 'N' indicates the number of samples that do not satisfy the intra-class closeness. We observed that with a higher \(\theta\) value, the number of samples satisfying the intra-class closeness decreases and the number of samples transferred to human experts increases. Conversely, with a lower \(\theta\) value, the proportion of the samples that the machine classifies with the KNN classifier will increase, but it may have the risk of a lower accuracy. For example, with \(\theta=3\), there were 45 out of 271 inter-class outliers that were additionally classified with the 1-NN classifier because it satisfied the intra-class closeness, while the remaining 226 samples were transferred to human experts. Of the additional 45 samples, 44 samples were classified correctly with an accuracy of 97.8%. However, with \(\theta=1.2\), only 186 samples satisfied the intra-class closeness, and, among them, only 161 were classified correctly with an accuracy of 86.6%, and the remaining 85 samples were transferred to human experts. To summarize, a higher \(\theta\) provided higher accuracy in general. However, a higher \(\theta\) also caused several target malwares not to satisfy the relative closeness, which might result in time and other expenses for the classification transferred to human experts. Table 3 summarizes the classification accuracy and proportion of classified samples after applying the two metrics, intra- and inter-class closeness. First, 1819 samples were classified accurately with an accuracy of 93.7% using the KNN classifier. After applying the inter-class closeness, 1613 out of 1670 samples could be classified accurately with an accuracy of 96.6%. This significant accuracy could be achieved by postponing the 271 inter-class outliers. Then, with the intra-class closeness applied to the 271 inter-class outliers, additional samples can be classified with the KNN classifier depending on the value of \(\theta\). When \(\theta\) decreases, we can observe the increasing ratio of classified samples and a slight decline in the overall accuracy. We found that \(\theta=1.5\) shows the most balance between the accuracy and postponing rate.