## 5 Experimental Studies
### Influence of Dataset Size and Program Embedding on Classifier Quality
When using a small dataset of source codes, RF and SVM-based classifiers perform best with embeddings that utilize structural information about programs, such as Markov chains, code2 vec, AST node-type histograms, and bytecode histograms, as shown in Figure 19 and in Table 4. The simple KNN classifier shows good results when used together with first-order Markov chains or together with histogram-based embeddings. Moreover, first-order and second-order AST-based Markov chains, as well as histograms of AST node types used together with RF, show the best classification quality compared to other pairs of a program embedding and this classifier. The surprisingly low quality of MLP-based classifiers used together with histogram-based embeddings may indicate that further hyperparameter tuning is required for MLP when used with histograms. The data shown in Figure 20 and Table 5 compare the classification quality for different pairs of an embedding and a classifier for a dataset containing 1000 samples. The results obtained for 1000 programs shown in Figure 20 and Table 5 are partially similar to the results obtained for 100 programs (see Figure 19). According to Figures 19 and 20, second-order AST-based Markov chains perform significantly worse when used with KNN when compared to first-order Markov chains. Both first-order and second-order Markov chains achieve the best _Accuracy_ (13), _Precision_ (14), _Recall_ (15), and _F1 Score_ (16) when used with RF and MLP. As shown in Figures 19 and 20, increasing the order of an AST-based Markov chain gives no noticeable benefits, as first-order and second-order Markov chains show similar performance when used with RF. In the case of KNN, increasing the order of a Markov chain can decrease classifier quality. According to Figure 20 and Table 5, the token-based embeddings perform significantly better with the increase in the dataset size, leaving behind code2 vec-based embeddings used together with RF and histogram-based embeddings used with MLP. Token context in token-based embeddings is represented only by the surrounding tokens, and hence such embeddings are limited by the lack of the structural information of a code snippet. Programs implementing different algorithms have differences in structure that might not be taken into account in token-based embeddings. Additionally, token contexts that appear in programs implementing different algorithms introduce additional noise that might lead to the degradation of the quality of classifiers. The influence of such noise is especially noticeable on small datasets, with a small number of program examples that solve the same task and have many common token contexts. Figure 20: Classification quality for different pairs of a program embedding and a classifier for a small-sized dataset of source codes containing 1000 programs from the original [19] dataset. The embeddings based on first-order Markov chains tend to be least sensitive to the used classifier, showing good results in the sense of (13)-(16) with all of the considered classifiers, according to Figures 19 and 20. Aiming to either confirm or disprove this, we numerically assessed the sensitivity of different approaches to source code transformation into vector-based representations by analyzing the collected data presented in Tables 4 and 5; the results of assessments are presented in Section 5.3. Aiming to statistically verify the observed superiority of program embeddings that are based on first-order Markov chains, we applied the Wilcoxon signed rank test [52] to the obtained quality assessments of different classifiers. In order to obtain statistically representative results, we repeated the evaluation of each pair of a classifier and an embedding 10 times; each time the datasets containing 100 and 1000 programs were split into 5 blocks, and the classifiers were assessed on 5 different parts of each dataset (see Figure 13). Hence, each of the obtained classifier quality distributions contained 50 values in total.