## 5 Experimental Studies
### Influence of Dataset Size and Program Embedding on Classifier Quality
\begin{table}
\begin{tabular}{c c} \hline \hline
**Algorithm** & **Parameters** \\ \hline KNN & 4 neighbors, distance-based neighbor weighting. \\ SVM & Radial-basis function kernel, C = 30, one-vs-one strategy. \\ RF & Max. tree depth is 40, max. tree count is 300, gini criterion. \\ MLP & 100 hidden layer neurons, ReLU activation, Adam optimizer with \\  & learning rate = 0.001, \(\beta_{1}\) = 0.9, \(\beta_{2}\) = 0.999, \(\epsilon=10^{-8}\). \\ word2 vec & 100 vector components, window size = 5, 5 negative samples, \\  & \(\alpha\) = 0.025, 5 epochs. \\ code2 vec & 384 vector components, dropout keep rate = 0.75, 20 epochs. \\ PathMiner & ANTLR Python parser, maximum path length = 30, maximum path \\  & width 15, maximum path contexts per entity = 300. \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Parameters of source code embedding, utility, and classification algorithms. \(\mu+3\sigma\) and \(\mu-3\sigma\) according to the three-sigma rule [50], where \(\mu\) denotes the mean (7) and \(\sigma\) denotes the standard deviation (8) computed based on the observations obtained using 5-fold cross-validation (see Figures 13 and 14). According to Figure 15, AST-based and bytecode-based program embeddings outperform token-based embeddings on small datasets containing less than 500 programs. The situation changes for medium-sized datasets containing around 1000 programs, where word2 vec shows the same results as code2 vec and second-order Markov chains. The use of AST-based histograms, first-order Markov chains, and bytecode-based histograms leads to the best KNN classifier quality, although these are the simplest embeddings among the considered ones that do not involve neural network training or hyperparameter tuning. The results are consistent with [28; 51], where the authors show that low-resource algorithms have the potential to outperform deep neural networks. Figure 16 shows the influence of dataset size and program embedding on the quality of classification results obtained with the SVM algorithm. As shown in Figure 16, AST-based and bytecode-based embeddings again outperform token-based embeddings on small-sized and medium-sized datasets. Source code  
Figure 16: The influence of dataset size and program embedding on the SVM classifier quality. Figure 15: The influence of dataset size and program embedding on the KNN classifier quality. vector representations that were obtained using first-order Markov chains and bytecode histograms perform best on small datasets when used with SVM. The increase in the dataset size leads to the increase in token-based embeddings quality. Figure 17 compares the embeddings used together with an RF-based classifier. According to Figure 17, first-order and second-order Markov chains, and AST-based and bytecode-based histograms, outperform word2 vec and code2 vec on small-sized and medium-sized datasets when used with an RF-based classifier in the considered task detection problem. AST-based histograms and Markov chains require fewer source code examples for obtaining high RF classifier quality on small datasets containing around 100 programs when compared to bytecode-based histograms. This may indicate that the neural-network-based continuous vectors are harder for RF to cope with, and AST-based features are of high importance in the task detection problem. Figure 18 compares the embeddings used together with an MLP-based classifier. According to Figure 18, AST-based Markov chains show the best results among the considered approaches to source code transformation into vectors used together with an MLP-based classifier. In contrast to RF (see Figure 17) and KNN (see Figure 15), AST-based and bytecode-based histograms perform significantly worse with MLP, and worse than  
Figure 17: The influence of dataset size and program embedding on the RF classifier quality. Figure 18: The influence of dataset size and program embedding on the MLP classifier quality. token-based embeddings on medium-sized datasets.