# Machine Learning-Based Security Pattern Recognition Techniques for Code Developers
## 3 Experimental Results
### Feature Importance  
To understand which elements (features) of the data flow and control flow sections of the Intermediate Representation contribute more to the preservation of the CWE patterns, we have calculated the feature performance per each CWE, using the Gini Importance built-in within the Random Forest algorithm provided by scikit-learn, and aggregated the results, as displayed in Figure 4.  
We observe that data flow section of the Intermediate Representation contributes strongly to the preservation of CWE patterns. Tokens captured by the control flow have similar importance. The feature importance is dependent on the vulnerability itself (CWE) and also to the average size of source code snippets (some CWE may be captured in less lines of code than others).  
Features which appear in the first part of the control flow and data flow vector have, in general, higher importance. One reason is the behavior of NIST suite developers who included most of the CWE relevant information (tokens, identifiers) in the first part of the code snippet. Another reason is the differences in code size between CWEs (some CWE patterns are included in fewer lines of code). Feature importance has been calculated for C/C++ code, when the techniques of dataset preparation is based on separation per code  
\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline
**ML Algorithm** & \multicolumn{3}{c}{**Random Forest**} & \multicolumn{3}{c}{**Decision Tree**} & \multicolumn{3}{c}{**K-Nearest Neighbors**} \\ \hline \hline
**Number of Vulnerable** & \multirow{2}{*}{**F1**} & \multirow{2}{*}{**Recall**} & \multirow{2}{*}{**Precision**} & \multirow{2}{*}{**F1**} & \multirow{2}{*}{**Recall**} & \multirow{2}{*}{**Precision**} & \multirow{2}{*}{**F1**} & \multirow{2}{*}{**Recall**} & \multirow{2}{*}{**Precision**} \\
**Samples C/C++** & & & & & & & & & \\ \hline \(>\)5000 & 0.778 & 0.899 & 0.691 & 0.754 & 0.778 & 0.734 & 0.776 & 0.764 & 0.790 \\ \(>\)2000 & 0.668 & 0.782 & 0.472 & 0.578 & 0.609 & 0.552 & 0.631 & 0.638 & 0.626 \\ \(>\)1000 & 0.694 & 0.886 & 0.581 & 0.682 & 0.710 & 0.658 & 0.726 & 0.729 & 0.727 \\ \(>\)500 & 0.651 & 0.899 & 0.523 & 0.648 & 0.676 & 0.625 & 0.694 & 0.711 & 0.687 \\ \(>\)100 & 0.602 & 0.894 & 0.474 & 0.605 & 0.625 & 0.591 & 0.663 & 0.700 & 0.644 \\ \(>\)0 & 0.506 & 0.838 & 0.383 & 0.493 & 0.513 & 0.495 & 0.582 & 0.658 & 0.555 \\ \hline
**Number of Vulnerable** & \multirow{2}{*}{**F1**} & \multirow{2}{*}{**Recall**} & \multirow{2}{*}{**Precision**} & \multirow{2}{*}{**F1**} & \multirow{2}{*}{**Recall**} & \multirow{2}{*}{**Precision**} & \multirow{2}{*}{**F1**} & \multirow{2}{*}{**Recall**} & \multirow{2}{*}{**Precision**} \\
**Samples Java** & & & & & & & & & \\ \hline \(>\)5000 & 0.953 & 0.984 & 0.923 & 0.955 & 0.990 & 0.922 & 0.929 & 0.980 & 0.882 \\ \(>\)2000 & 0.926 & 0.958 & 0.896 & 0.929 & 0.972 & 0.890 & 0.899 & 0.960 & 0.845 \\ \(>\)1000 & 0.920 & 0 sniippet function and CWE (more vulnerability context may be reflected in the features which are not directly connected to the CWE).  
As a conclusion, the data flow has an important contribution to the CWE pattern preservation, meaning that identifiers and their relative position within the code are not less important than the programming language keywords and their position in the same code. In this context, the elasticity of CWE patterns (e.g., different sizes of control and data flow vectors with non-null elements for the same code snippet or function) makes our approach of splitting the code into control and data flow both practical and CWE remanent.