# Security Monitoring during Software Development:
## 3 Methodology
### Configuration of the Core Security Monitoring Mechanisms
#### 3.3.2 Vulnerability Prediction
The k-fold cross-validation process is a method of dividing the whole training dataset into k folds recursively considering, each time, one different fold as the testing one. We compute the predictive accuracy of the models for k folds, and after the process is completed, we compute the average accuracy. This way, we avoid putting data bias on the model. For the evaluation of the efficiency of the models, we do not use only the accuracy metric but also precision, recall, F\({}_{1}\)-score, and F\({}_{2}\)-score in order to obtain a complete estimation of the predictive power of the produced models, considering True Positives (TPs), True Negatives (TNs), False Positives (FPs), and False Negatives (FNs). The formulas of the evaluation metrics, which were computed using the scikit-learn ([[https://scikit-learn.org/stable/modules/model_evaluation.html](https://scikit-learn.org/stable/modules/model_evaluation.html)]([https://scikit-learn.org/stable/modules/model_evaluation.html](https://scikit-learn.org/stable/modules/model_evaluation.html)), accessed on 27 April 2023) python library, are provided in the equations below:  
\[Accuracy=\frac{TP+TN}{TP+FP+TN+FN} \tag{1}\]  
\[Precision=\frac{TP}{TP+FP} \tag{2}\]  
\[Recall=\frac{TP}{TP+FN} \tag{3}\]  
\[F_{1}=\frac{2\times precision\times recall}{precision+recall} \tag{4}\]  
\[F_{2}=\frac{5\times precision\times recall}{4\times precision+recall} \tag{5}\]  
The architecture of the best model is presented in Table 9. Table 10 contains the values of the utilized evaluation metrics of our best model, which is based on the combination of CNN and word2 vec algorithms. For reasons of completeness, we provide also the results of the BoW method, which uses an ensemble ML algorithm called Random Forest [60]. As can be seen by Table 10, the representation method of sequences of tokens seems to be superior from the BoW method. Although BoW achieves a higher precision, sequences of tokens succeed higher scores in all the other evaluation metrics. Hence, the sequences of tokens method, which uses CNN and word2 vec, proved to be much more efficient in identifying vulnerabilities, even if it produces slightly more FPs. In fact, the F\({}_{2}\)-score, which is considered more important for the case of vulnerability prediction as it considers both Recall and Precision, but puts more emphasis on Recall, of the CNN with word2 vec model was found to be much higher compared to the score of the model that utilized the BoW approach. \begin{table}
\begin{tabular}{l l} \hline \hline
**Hyperparameter Name** & **Value** \\ \hline Number of Layers & 3 (Embedding–Convolutional–Dense) \\ Number of Convolutional Layers & 1 \\ Embedding Size & 300 \\ Number of Filters & 128 \\ Kernel Size & 5 \\ Pooling & global max pooling \\ Weight Initialization Technique & glorot uniform (Xavier) \\ Learning Rate & 0.01 \\ Gradient Descent Optimizer & Adam \\ Batch Size & 64 \\ Activation Function & relu \\ Output Activation Function & sigmoid \\ Loss Function & binary cross-entropy \\ \hline \hline \end{tabular}
\end{table} 
Table 9: The selected hyperparameters of the CNN model.