## 5 Experimental Studies
### Influence of Dataset Size and Program Embedding on Classifier Quality
According to Tables 6 and 7, first-order Markov-chain-based program embeddings outperform second-order Markov-chain-based embeddings on datasets containing 100 and 1000 programs when used together with KNN and MLP. No statistically significant differences were found between the forementioned embeddings when they were used together with RF and MLP. This may indicate that increasing the order of a Markov chain offers no benefits, and first-order AST-based Markov chains are most suitable for practical applications. First-order Markov chains outperform histograms of AST node types and histograms of assembly language instruction opcodes when the embeddings are used with RF and MLP. Histogram-based embeddings outperform Markov chains on the smallest dataset when used together with the distance-based KNN classifier. As shown in Tables 6 and 7, program embeddings based on Markov chains outperform embeddings based on word2 vec and code2 vec. The discovered differences among the considered pairs of a program embedding and a classifier might be caused by the differences in the structure and count of components in the continuous vectors representing programs. Table 8 lists the total component count, non-zero component count, and zero component count in a sample vector for each of the considered embeddings for a dataset containing 100 programs. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Classifier** & \multicolumn{2}{c}{**KNN**} & \multicolumn{2}{c}{**SVM**} & \multicolumn{2}{c}{**RF**} & \multicolumn{2}{c}{**MLP**} \\ \hline
**Embedding** & **Sign** & \(\mathbf{p}\)**-Value** & **Sign** & \(\mathbf{p}\)**-Value** & **Sign** & \(\mathbf{p}\)**-Value** & **Sign** & \(\mathbf{p}\)**-Value** \\ \hline word2 vec & + & 0.004 & + & 0.004 & + & 0.004 & + & 0.004 \\ Second-order AST-based & + & 0.011 & = & 0.250 & = & 0.916 & = & 0.652 \\ Markov chain & = & 0.820 & = & 0.359 & + & 0.004 & + & 0.008 \\ Histograms for bytecode & – & 0.019 & = & 0.359 & = & 0.207 & + & 0.004 \\ Histograms for AST & – & 0.004 & = & 0.164 & = & 0.498 & + & 0.004 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Results of the Wilcoxon signed rank test applied to program embeddings based on first-order Markov chains and all other embeddings used with different classification algorithms trained to solve the task classification problem on a dataset containing 100 programs. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Classifier** & \multicolumn{2}{c}{**KNN**} & \multicolumn{2}{c}{**SVM**} & \multicolumn{2}{c}{**RF**} & \multicolumn{2}{c}{**MLP**} \\ \hline
**Embedding** & **Sign** & \(\mathbf{p}\)**-Value** & **Sign** & \(\mathbf{p}\)**-Value** & **Sign** & \(\mathbf{p}\)**-Value** & **Sign** & \(\mathbf{p}\)**-Value** \\ \hline word2 vec & + & 0.004 & + & 0.004 & + & 0.004 & + & 0.004 \\ Second-order AST-based & + & 0.004 & = & 0.652 & = & 0.150 & + & 0.027 \\ Markov chain & + & 0.004 & + & 0.004 & + & 0.004 & + & 0.004 \\ Histograms for bytecode & = & 0.570 & = & 0.359 & + & 0.027 & + & 0.004 \\ Histograms for AST & = & 0.426 & = & 0.074 & + & 0.042 & + & 0.004 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Results of the Wilcoxon signed rank test applied to program embeddings based on first-order Markov chains and all other embeddings used with different classification algorithms trained to solve the task classification problem on a dataset containing 1000 programs. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Program Embedding** & **Total Components** & **Non-Zero** & **Zero** \\ \hline word2 vec & 100 & 100 & 0 \\ First-order Markov chain & 4096 & 17 & 4079 \\ Second-order Markov chain & 35,721 & 23 & 35,698 \\ code2 vec & 384 & 384 & 0 \\ Histograms for bytecode & 73 & 15 & 58 \\ Histograms for AST & 75 & 19 & 56 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Total component count and non-zero component count in different embeddings. Program vectors representing Markov chains are sparse.