## 3 Program Text Vectorization
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Replacements for 1–11 Task Types** & \multicolumn{3}{c}{**Replacements for 6–11 Task Types**} \\ \hline
**Replaceable Vertex** & **Replacement** & **Replaceable Vertex** & **Replacement** \\ \hline Constant, Attribute & Name & List, Tuple, Set & List \\ BoolOp, Call, UnaryOp, BinOp & Op & Lambda, JoinedStr, FormattedValue & Name \\ Lt, LtE & Less & Pass, Break, Continue & None \\ Gt, GtE & Greater & ExceptHandler & Try \\ AugAssign, AnAssign & Assign & IfExp & If \\ match\_case, MatchStar, MatchAs, MatchOr, & For, While, ListComp, SetComp, & \\ MatchSingleton, MatchSequence, & MatchValue & DictComp, GeneratorExp, & Loop \\ MatchMapping, MatchClass & & comprehension & \\ UAdd & Add & & \\ USub & Sub & & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Replacements of AST node types while building Markov chain state transition graphs. Figure 3: Markov chain state transition graphs for programs that employ different approaches to solve automatically generated tasks of type 6 (see Table 1, see Figure 1a): (**a**) solution using conditional operators; (**b**) solution using pattern matching; (**c**) solution using dictionaries. In the DTA analytics module, programs solving automatically generated tasks are vectorized all at once according to Algorithm 1, which only requires a set of program texts as input. After applying an unsupervised machine learning algorithm to the vectorized data, the DTA analytics module outputs a marked-up dataset. In that dataset, every \(i\)-th object is represented by a \(\overrightarrow{v}_{i}\in\mathbb{R}^{m}\) vector, where \(m=|H|^{2}\), and \(H\) denotes the set containing all different AST node types that occur in all Markov chains obtained while vectorizing a particular set of program texts. Every \(i\)-th object has an associated class label \(y_{i}\in\mathcal{Y}\), where \(\mathcal{Y}\) is the set containing all possible classes discovered for a given dataset. Algorithm 1 is only able to vectorize a set of program texts. However, vectorization of a single program text is required for unseen data classification. In order to vectorize a single program text \(p_{i}\in P\), Algorithm 2 is used. Algorithm 2 uses the \(H\) set while building a weighted adjacency matrix; the \(H\) set is obtained from Algorithm 1 and saved to disk before applying Algorithm 2. ```
Input:\(p_{i}\)-a program text, \(H\)-a set with all different node types that exist in Markov chains of programs solving tasks of the same type as \(p_{i}\), this set is obtained by Algorithm 1. 1. Define\(R=\textit{[Import, Load, Store, alias, arguments, arg, Module, keyword]}\). 2. Construct an AST \(a_{i}\) for \(p_{i}\) using Python standard library (see Figure 2). 3. Remove nodes belonging to the \(R\) set from \(a_{i}\). 4. Construct an AST \(a_{i}\) for \(p_{i}\) using Python standard library (see Figure 2). 5. Remove nodes belonging to the \(R\) set from \(a_{i}\). 6. Replace nodes in \(a_{i}\) according to Table 2.
7. Construct Markov chain state transition graph \(g_{i}\) for \(a_{i}\) (see Figure 3). 8. \(G\gets G\cup\{g_{i}\}\). 9. End loop. 10. Define the set \(H\) with all different node types existing in graphs from \(G\). 11. Define the set of vector representations of program texts \(V=\varnothing\). 12. For each Markov chain state transition graph \(g_{i}\in G\)do:
13. Construct an adjacency matrix \(\mathbf{m}_{i}\in\mathbb{R}^{|H|\times|H|}\) for the weighted graph \(g_{i}\). 14. Reshape the \(\mathbf{m}_{i}\) matrix to vector \(\overrightarrow{v}_{i}\in\mathbb{R}^{m}\), where \(m=|H|^{2}\). 15. \(V\gets V\cup\left\{\overrightarrow{v}_{i}\right\}.\)
16. End loop. 17. Return the set of vector representations of program texts \(V\) and the set \(H\). ```  
**Algorithm 2** Vectorization of a single program text \(p_{i}\) based on Markov chains. To sum up, the main difference between Algorithms 1 and 2 is that Algorithm 1 can only vectorize a set of program texts solving automatically generated tasks of a given type.