# Security Monitoring during Software Development:
## 3 Methodology
### Configuration of the Core Security Monitoring Mechanisms
#### 3.3.2 Vulnerability Prediction
During the developing of our vulnerability prediction model, we employed and compared both source code representation methods: Bag of Words (BoW) and sequences of tokens. We used these two forms of the constructed dataset to train and evaluate machine learning (ML) models capable of identifying vulnerabilities in the source code. As depicted in Figure 15, BoW is a numerical representation of each source code file putting particular emphasis on the existing tokens and the frequency in which they appear in the code. In particular, in the BoW method each source code file is represented by a numerical vector, the cells of which correspond to the frequency of each token (i.e., instruction/keyword), i.e., the number of its occurrences in the corresponding source code file. On the other hand, sequences of tokens is a way of representing software components (e.g., files) as sequences of the words that exist in the specific component, taking into consideration the words' position in each sequence. In this case, for the numerical representation of the tokens, word embedding techniques are applied. In particular, we examined the capacity of the word2 vec [53] and fastText algorithms [57], and we compared each by conducting an empirical evaluation. More specifically, word2 vec is a ML model proposed by [PERSON] et al. [53] that can learn word embedding vectors by predicting the context of the words within a given text corpus. The generated vectors, which are numerical representations of the words, are positioned in such a way that similar words are closer together in the vector space. This way, word embedding representations enhance the attempt of the text-mining-based models to capture syntactic and semantic patterns through the text (i.e., through the source code in VP case). An extension of word2 vec is fastText, proposed by [PERSON] et al. [57]. It introduces sub-word information into the word embeddings. In other words, it considers tokens as smaller sub-word units and produces embedding vectors for them. This way, in contrast with word2 vec, fastText can handle out-of-vocabulary words, since it can represent unseen words based on the embeddings of the sub-words that reside in these unknown words. Regarding the ML models that we employed for the construction of our VPM, we focused on deep learning architectures such as Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) units which is an RNN variant, and Convolutional Neural Networks (CNNs). The aforementioned architectures are able to receive and learn sequential data as the sequences of the tokens that exist in the source code. In particular, LSTM is specifically designed for learning sequential data, capturing long-term dependencies by employing memory units and gates [58]. CNNs are commonly utilized in image processing tasks, but one-dimension (1-D) convolutional layers can be easily utilized for learning sequences. To find the optimal model of each examined deep learning architecture, during the prediction model generator phase (see Figure 14), we applied hyperparameter tuning using a grid search process [59]. To develop the models, we utilized the Keras ([[https://keras.io/api/models/sequential/](https://keras.io/api/models/sequential/)]([https://keras.io/api/models/sequential/](https://keras.io/api/models/sequential/)), accessed on 27 April 2023) library of the TensorFlow framework ([[https://www.tensorflow.org/](https://www.tensorflow.org/)]([https://www.tensorflow.org/](https://www.tensorflow.org/)), accessed on 27 April 2023). The training and evaluation processes took place on the CUDA platform ([[https://developer.nvidia.com/cuda-toolkit](https://developer.nvidia.com/cuda-toolkit)]([https://developer.nvidia.com/cuda-toolkit](https://developer.nvidia.com/cuda-toolkit)), accessed on 27 April 2023) installed on an NVIDIA 3060 RTX GPU. For the empirical evaluation of the aforementioned models and techniques, we employed an evaluation scheme based on the process of cross-validation.