## 5 Experimental Studies
### Influence of Dataset Size and Program Embedding on Classifier Quality  
As described in Section 5.1, specialized metrics are used for classification quality assessment on testing parts of the dataset. In a binary classification problem, a set of possible answers \(Y=\{-1,+1\}\), where \"\(-1\)\" denotes the negative class, and \"\(+1\)\" denotes the positive class (see Section 3 for a classification problem definition). Binary classifiers can make two types of mistakes by either erroneously assigning an object to class \"\(+1\)\", which is known as _false positive_ (_FP_), or erroneously assigning an object to class \"\(-1\)\", which is called _false negative_ (_FN_). The correct answer given by a classifier can be either _true positive_ (_TP_) or _true negative_ (_TN_). For a classifier \(a:\mathbb{X}\to Y\),  
\[FP=|\{(x_{i},y_{i})\in U:a(x_{i})=+1\wedge y_{i}=-1\}|, \tag{9}\]  
\[FN=|\{(x_{i},y_{i})\in U:a(x_{i})=-1\wedge y_{i}=+1\}|, \tag{10}\]  
Figure 14: The source code classification framework used in the experiments. Figure 13: \(k\)-fold cross-validation for evaluating classifier performance. \[TP=|\{(x_{i},y_{i})\in U:a(x_{i})=+1\wedge y_{i}=+1\}|, \tag{11}\]  
\[TN=|\{(x_{i},y_{i})\in U:a(x_{i})=-1\wedge y_{i}=-1\}|, \tag{12}\]  
where \(U\) is the dataset (see Section 3) containing \(s\) pairs \((x_{i},y_{i})\), \(x_{i}\in X\) is the \(i\)-th object belonging to the set of possible objects that can be passed to a classifier \(a:\mathbb{X}\to Y\), and \(y_{i}\in Y\) is the answer associated with the \(i\)-th object \(x_{i}\). In a multiclass classification problem, the widely used quality assessment metrics include macro-_Accuracy_, macro-_Precision_, macro-_Recall_, and macro-_F1 Score_[15; 32]. In these metrics, _FP_ (9), _FN_ (10), _TP_ (11), and _TN_ (12) values are computed separately for each class \(k\), treating all other classes as negative, and then combined according to:  
\[Accuracy=\frac{1}{c}{\sum_{k=1}^{c}}\bigg{(}\frac{TP_{k}+TN_{k}}{TP_{k}+TN_{k} +FP_{k}+FN_{k}}\bigg{)}, \tag{13}\]  
\[Precision=\frac{1}{c}{\sum_{k=1}^{c}}\bigg{(}\frac{TP_{k}}{TP_{k}+FP_{k}} \bigg{)}, \tag{14}\]  
\[Recall=\frac{1}{c}{\sum_{k=1}^{c}}\bigg{(}\frac{TP_{k}}{TP_{k}+FN_{k}}\bigg{)}, \tag{15}\]  
\[F1\ Score=2\cdot\bigg{(}\frac{Precision\ \cdot\ Recall}{Precision+Recall} \bigg{)}, \tag{16}\]  
where \(c\) denotes the total class count, \(TP_{k}\), \(TN_{k}\), \(FP_{k}\), and \(FN_{k}\) denote (9)-(12) values for the \(k\)-th class versus all other classes, the class \(k\) is treated as positive, and other classes are treated as negative; these metrics take class imbalance into account [32]. The parameters of the considered classification algorithms are listed in Table 3, alongside the parameters of word2 vec Gensim implementation (see Section 4.1), code2 vec implementation [9], and PathMiner (see Section 4.2). The approaches to source code embedding into vector space that are based on first-order and second-order Markov chains (see Section 4.3), as well as the histogram-based approaches (see Sections 4.4 and 4.5), do not have any hyperparameters. The parameters that are not listed in Table 3 are initialized with the default values recommended by [9; 38; 49]. **Question 1 (RQ1). **_How does the size of the dataset and the selected approach to source code transformation into vector-based representation affect the quality of different classification algorithms in the task detection problem?_  
Aiming to determine how the size of the dataset and the selected approach to source code embedding into vector space influence the cross-validated (see Figures 13 and 14) quality in the sense of (13)-(16) of the KNN classifier, we obtained the plots shown in \[FN=|\{(x_{i},y_{i})\in U:a(x_{i})=-1\wedge y_{i}=+1\}|, \tag{10}\]0 The areas filled with different colors in Figure 15 denote the range limited by