## 5 Experimental Studies
### Influence of Dataset Size and Program Embedding on Classifier Quality
This may indicate that additional hyperparameter tuning is required when using MLP with histograms. Aiming to determine which pairs of an embedding and a classifier perform best on a small dataset containing only 100 programs, we obtained the plots shown in Figure 19.  
The line segments shown in Figure 19 denote intervals limited by \(\mu+\sigma\) and \(\mu-\sigma\), where \(\mu\) denotes the mean (7) and \(\sigma\) denotes the standard deviation (8) computed based on the observations obtained from 5-fold cross-validation (see Figures 13 and 14). The data visualized in Figure 19 is also presented in Table 4. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Embedding** & **Classifier** & _Accuracy (13)_ & _Precision (14)_ & _Recall (15)_ & _F1 Score (16)_ \\ \hline \multirow{4}{*}{\begin{tabular}{c} word2 vec \\ (See Section 4.1) \\ \end{tabular} } & KNN & \(39.0\pm 10.7\) & \(30.3\pm 8.3\) & \(37.2\pm 7.9\) & \(31.8\pm 7.5\) \\  & SVM & \(25.0\pm 3.2\) & \(11.7\pm 6.2\) & \(21.9\pm 4.1\) & \(13.4\pm 5.8\) \\  & RF & \(65.0\pm 6.3\) & \(59.0\pm 5.0\) & \(61.4\pm 5.4\) & \(57.8\pm 5.4\) \\  & MLP & \(19.0\pm 7.3\) & \(5.1\pm 3.7\) & \(14.8\pm 5.3\) & \(6.8\pm 4.5\) \\ \hline \multirow{4}{*}{\begin{tabular}{c} First-order AST-based Markov chain \\ (See Section 4.3) \\ \end{tabular} } & KNN & \(81.0\pm 5.5\) & \(78.8\pm 5.3\) & \(78.2\pm 3.5\) & \(75.4\pm 5.1\) \\  & SVM & \(84.0\pm 7.3\) & \(78.6\pm 8.9\) & \(80.0\pm 8.0\) & \(77.0\pm 9.6\) \\  & RF & \(92.0\pm 4.0\) & \(88.2\pm 5.9\) & \(89.4\pm 5.3\) & \(87.8\pm 6.1\) \\  & MLP & \(86.0\pm 7.3\) & \(79.6\pm 10.5\) & \(82.0\pm 9.1\) & \(79.5\pm 10.0\) \\ \hline \multirow{4}{*}{\begin{tabular}{c} Second-order AST-based Markov chain \\ (See Section 4.3) \\ \end{tabular} } & KNN & \(70.0\pm 10.5\) & \(63.3\pm 12.7\) & \(66.1\pm 9.1\) & \(62.1\pm 11.3\) \\  & SVM & \(83.0\pm 6.8\) & \(75.4\pm 8.2\) & \(77.6\pm 6.5\) & \(75.0\pm 7.7\) \\  & RF & \(91.0\pm 5.8\) & \(88.2\pm 9.4\) & \(88.8\pm 8.1\) & \(87.1\pm 9.4\) \\  & MLP & \(88.0\pm 8.1\) & \(81.6\pm 10.8\) & \(83.9\pm 10.4\) & \(81.2\pm 11.4\) \\ \hline \multirow{4}{*}{\begin{tabular}{c} Histograms for Bytecode \\ (See Section 4.4) \\ \end{tabular} } & KNN & \(76.0\pm 9.7\) & \(70.2\pm 12.6\) & \(71.5\pm 10.8\) & \(68.6\pm 12.2\) \\  & SVM & \(81.0\pm 9.7\) & \(77.7\pm 7.8\) & \(79.3\pm 8.9\) & \(76.4\pm 9.3\) \\  & RF & \(74.0\pm 5.8\) & \(67.6\pm 7.3\) & \(71.2\pm 7.1\) & \(67.9\pm 8.2\) \\  & MLP & \(76.0\pm 5.8\) & \(69.9\pm 6.7\) & \(71.8\pm 5.9\) & \(68.4\pm 6.4\) \\ \hline \multirow{4}{*}{\
\begin{tabular}{c} Histograms for Bytecode \\ (See Section 4.4) \\ \end{tabular} } & KNN & \(84.0\pm 10.7\) & \(78.7\pm 12.5\) & \(80.3\pm 12.1\) & \(78.3\pm 12.7\) \\  & SVM & \(87.0\pm 9.3\) & \(83.6\pm 11.6\) & \(84.8\pm 10.3\) & \(83.6\pm 11.3\) \\  & RF & \(86.0\pm 7.3\) & \(81.6\pm 10.2\) & \(82.4\pm 9.3\) & \(80.6\pm 10.3\) \\  & MLP & \(60.0\pm 7.1\) & \(44.7\pm 8.1\) & \(53.3\pm 5.6\) & \(45.9\pm 7.3\) \\ \hline \multirow{4}{*}{
\begin{tabular}{c} Histograms for AST \\ (See Section 4.5) \\ \end{tabular} } & KNN & \(84.0\pm 6.6\) & \(82.2\pm 5.8\) & \(81.9\pm 6.5\) & \(79.8\pm 6.5\) \\  & SVM & \(86.0\pm 5.8\) & \(83.1\pm 6.6\) & \(83.5\pm 6.6\) & \(81.4\pm 6.7\) \\ \cline{1-1}  & RF & \(92.0\pm 3.0\) & \(\mathbf{88.2\pm 4.9}\) & \(\mathbf{89.4\pm 4.3}\) & \(\mathbf{87.8\pm 5.1}\) \\ \cline{1-1}  & MLP & \(52.0\pm 6.8\) & \(34.3\pm 5.5\) & \(43.1\pm 4.1\) & \(35.6\pm 5.1\) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Classification quality for different pairs of a program embedding and a classifier assessed using different metrics for a small-sized dataset of source codes containing 100 programs. The top 3 best classifier quality scores for each metric are highlighted in bold. Figure 19: Classification quality for different pairs of a program embedding and a classifier for a small-sized dataset of source codes containing 100 programs from the original [19] dataset.