## 5 Experimental Studies
### Influence of Dataset Size and Program Embedding on Classifier Quality
The distribution obtained for first-order Markov-chain-based program embeddings was compared to every other program embedding studied in this research. According to the null hypothesis \(H_{0}\), the two compared distributions had no statistically significant differences [52]. The \(p\)-value was set to 0.05. The obtained results are presented in Table 6 for a dataset containing 100 programs (see also Figure 19). Table 7 lists the results for a dataset containing 1000 programs (see also Figure 20). The \"=\" sign in Tables 6 and 7 denotes that no statistically significant differences exist between the compared distributions, the \"+\" sign denotes that first-order Markov-chain-based embedding outperforms the other embedding, and the \"-\" sign denotes that first-order Markov-chain-based embedding leads to worse classifier quality. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Embedding** & **Classifier** & _Accuracy (13)_ & _Precision (14)_ & _Recall (15)_ & _F1 Score (16)_ \\ \hline \multirow{4}{*}{word2 vec} & KNN & 92.3 \(\pm\) 3.1 & 92.6 \(\pm\) 2.8 & 92.1 \(\pm\) 3.1 & 92.2 \(\pm\) 3.0 \\  & SVM & 92.5 \(\pm\) 2.3 & 92.7 \(\pm\) 2.3 & 92.3 \(\pm\) 2.3 & 92.3 \(\pm\) 2.3 \\ (See Section 4.1) & RF & 93.6 \(\pm\) 1.6 & 93.8 \(\pm\) 1.5 & 93.5 \(\pm\) 1.6 & 93.5 \(\pm\) 1.6 \\  & MLP & 91.0 \(\pm\) 1.8 & 91.3 \(\pm\) 2.1 & 90.7 \(\pm\) 1.8 & 90.8 \(\pm\) 1.8 \\ \hline \multirow{4}{*}{First-order AST-based Markov chain (See Section 4.3)} & KNN & 95.7 \(\pm\) 1.9 & 96.2 \(\pm\) 1.7 & 95.7 \(\pm\) 1.9 & 95.7 \(\pm\) 1.9 \\  & SVM & 96.7 \(\pm\) 0.7 & 97.0 \(\pm\) 0.7 & 96.7 \(\pm\) 0.6 & 96.7 \(\pm\) 0.6 \\ (See Section 4.3) & RF & **97.8 \(\pm\) 0.9** & **98.0 \(\pm\) 0.8** & **97.8 \(\pm\) 0.9** & **97.8 \(\pm\) 0.9** \\  & MLP & **97.9 \(\pm\) 1.2** & **98.0 \(\pm\) 1.2** & **97.9 \(\pm\) 1.2** & **97.9 \(\pm\) 1.2** \\ \hline \multirow{4}{*}{Second-order AST-based Markov chain (See Section 4.3)} & KNN & 92.5 \(\pm\) 2.3 & 94.0 \(\pm\) 1.4 & 92.4 \(\pm\) 2.4 & 92.3 \(\pm\) 2.4 \\  & SVM & 97.0 \(\pm\) 1.7 & 97.3 \(\pm\) 1.5 & 96.9 \(\pm\) 1.8 & 97.0 \(\pm\) 1.7 \\ \cline{1-1}  & RF & **98.2 \(\pm\) 1.3** & **98.3 \(\pm\) 1.2** & **98.2 \(\pm\) 1.3** & **98.2 \(\pm\) 1.3** \\ \cline{1-1}  & MLP & 97.5 \(\pm\) 0.8 & 97.7 \(\pm\) 0.8 & 97.5 \(\pm\) 0.7 & 97.5 \(\pm\) 0.7 \\ \hline \multirow{4}{*}{code2 vec (See Section 4.2)} & KNN & 92.1 \(\pm\) 1.3 & 92.5 \(\pm\) 1.4 & 92.0 \(\pm\) 1.3 & 92.0 \(\pm\) 1.4 \\  & SVM & 94.3 \(\pm\) 1.9 & 94.5 \(\pm\) 1.9 & 94.3 \(\pm\) 1.8 & 94.2 \(\pm\) 1.9 \\ (See Section 4.2) & RF & 90.9 \(\pm\) 2.5 & 91.4 \(\pm\) 2.3 & 90.3 \(\pm\) 2.7 & 90.4 \(\pm\) 2.6 \\  & MLP & 94.5 \(\pm\) 1.2 & 94.6 \(\pm\) 1.3 & 94.3 \(\pm\) 1.3 & 94.2 \(\pm\) 1.3 \\ \hline \multirow{4}{*}{Histograms for Bytecode (See Section 4.4)} & KNN & 96.1 \(\pm\) 0.9 & 96.3 \(\pm\) 0.9 & 96.1 \(\pm\) 0.9 & 96.1 \(\pm\) 0.9 \\  & SVM & 97.2 \(\pm\) 1.2 & 97.4 \(\pm\) 1.1 & 97.1 \(\pm\) 1.3 & 97.2 \(\pm\) 1.2 \\ (See Section 4.4) & RF & 97.0 \(\pm\) 1.0 & 97.2 \(\pm\) 0.8 & 97.0 \(\pm\) 1.1 & 97.0 \(\pm\) 1.0 \\  & MLP & 91.8 \(\pm\) 1.6 & 92.4 \(\pm\) 1.1 & 91.6 \(\pm\) 1.7 & 91.6 \(\pm\) 1.5 \\ \hline \multirow{4}{*}{Histograms for AST (See Section 4.5)} & KNN & 95.5 \(\pm\) 0.9 & 95.7 \(\pm\) 0.8 & 95.4 \(\pm\) 1.1 & 95.4 \(\pm\) 1.0 \\  & SVM & 96.6 \(\pm\) 1.4 & 96.7 \(\pm\) 1.3 & 96.5 \(\pm\) 1.4 & 96.5 \(\pm\) 1.4 \\ (See Section 4.5) & RF & 97.2 \(\pm\) 0.7 & 97.4 \(\pm\) 0.7 & 97.2 \(\pm\) 0.8 & 97.2 \(\pm\) 0.8 \\ \cline{1-1}  & MLP & 88.4 \(\pm\) 1.4 & 89.0 \(\pm\) 1.3 & 87.8 \(\pm\) 1.4 & 87.9 \(\pm\) 1.4 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Classification quality for different pairs of a program embedding and a classifier assessed using different metrics for a small-sized dataset of source codes containing 1000 programs. The top 3 best classifier quality scores for each metric are highlighted in bold.