## 5 Experimental Studies
### Sensitivity of Program Embeddings to the Used Classification Algorithm  
**Question 2 (RQ2). **_Which of the considered approaches to source code transformation into vector-based representation is least sensitive to the used classification algorithm in the task detection problem?_  
A program embedding that does not depend on a specific classification algorithm and works well with different classifiers has the potential to find more possible applications in different tasks and domains. For example, a simple KNN algorithm is used when labeled data are hard or impossible to obtain, or when a non-parametric classification technique is required [51]. RF, SVM, and MLP classifiers are used when high accuracy is required, and the specific classifier is typically chosen depending on the domain and data. In order to numerically determine which approach to source code embedding into vector space is least sensitive to the used classification algorithm in the considered task detection problem, we use the weighted standard deviation (weighted SD) as the measure of embedding sensitivity to the classification algorithm used:  
\[\overline{\sigma}(\mathbb{Q})=\sqrt{\sum_{j=1}^{n}(\mu(Q_{j})-\overline{\mu}( \mathbb{Q}))^{2}\frac{\sigma^{-1}(Q_{j})}{\sum_{i=1}^{n}\sigma^{-1}(Q_{i})}}, \tag{17}\]  
\[\overline{\mu}(\mathbb{Q})=\sum_{j=1}^{n}\mu(Q_{j})\frac{\sigma^{-1}(Q_{j})}{ \sum_{i=1}^{n}\sigma^{-1}(Q_{i})}, \tag{18}\]  
where \(n\) is the count of different classifier assessments in the \(\mathbb{Q}\) set obtained for a given program embedding, \(\mathbb{Q}=\left\{Q_{1},\ldots,Q_{j},\ldots,Q_{n}\right\};j=\overline{1,n}\) is the classifier number; \(Q_{j}\) is the set containing \(k\) classifier quality assessments obtained for the \(j\)-th classifier using \(k\)-fold cross-validation (see Figures 13 and 14); \(\mu(Q_{j})\) is the mean quality of the \(j\)-th classifier computed according to (7) based on \(k\)-fold cross-validation output \(Q_{j}\); and \(\sigma(Q_{j})\) is the standard deviation of the \(j\)-th classifier quality, computed according to (8) based on \(k\)-fold cross-validation output \(Q_{j}\). The weighted mean classifier quality (18) computed for a given program embedding is the overall quality of the program embedding, and the weighted SD (17) is the sensitivity of the program embedding to the used classification algorithm. In (17) and (18), the classifier quality assessments that have a small standard deviation \(\sigma(Q_{j})\) have a greater influence on the weighted SD value \(\overline{\sigma}(\mathbb{Q})\). For example, if we assess the sensitivity to the used classification algorithm of the embedding that is based on the first-order Markov chains, on a small dataset containing 100 programs using the _Accuracy_ (13) metric, for classifiers such as KNN, SVN, RF, and MLP,we respectively obtain the means \(\mu(Q_{1})=81.0\), \(\mu(Q_{2})=84.0\), \(\mu(Q_{3})=92.0\), \(\mu(Q_{4})=86.0\) (see Table 4), the standard deviations \(\sigma(Q_{1})=5.5\), \(\sigma(Q_{2})=7.3\), \(\sigma(Q_{3})=4.0\), \(\sigma(Q_{4})=7.3\), and the weighted SD value \(\overline{\sigma}(\mathbb{Q}_{1})\approx 4.45\), \(\mathbb{Q}_{1}=\{Q_{1},Q_{2},Q_{3},Q_{4}\}\). If we assess the sensitivity to the used classifier of the embedding that is based on second-order Markov chains on the same dataset according to (13) and Table 4, we obtain \(\overline{\sigma}(\mathbb{Q}_{2})\approx 7.36\). As \(\sigma(\mathbb{Q}_{1})<\sigma(\mathbb{Q}_{2})\), the first-order Markov-chain-based program embedding is less sensitive to the used classifier than the second-order Markov-chain-based embedding.