# PreNNsem: A Heterogeneous Ensemble Learning Framework for Vulnerability Detection in Software
## 3 Design of PreNNsem
### Representation Learning  
According to Hypothesis 2, common semantic characteristics can be learned from the context of vulnerabilities. Traditionally, the characteristics of manual definition are crucial to machine learning classification. They transform training data and then augment them with additional features to increase the efficacy of machine learning algorithms. However, with deep learning, we can start with raw data, as features will be automatically created by the neural network when it learns.  
In this section, we choose CNN and Long Short-Term Memory (LSTM) as the base deep learning model. Figure 3 shows the selected deep learning model used in this study. Firstly, in order to better learn the structure and semantics of the data, we used transfer learning to build the embedding layer for neural networks. Secondly, we sequentially combined three concatenated CNNs and one CNN as a network model. Thirdly, we added a one-dimensional max-pooling layer and a dropout layer for dimension reduction.  
What are the features learned by CNN? As shown in Figure 3, we represent a code segment of length \(n\) as:  
\[S=[v_{0},v_{1},...,v_{n}] \tag{3}\]  
Figure 3: Features of convolutional neural networks (CNNs).  
Figure 2: Overview of continuous bag-of-words (CBOW) models.  
where \(v_{i}\) is the \(i\)th word vector in the segment. A filter \(w_{h}\) is used to extract new features combined by the following \(h\) words. \(h\) is the size of the filter \(w_{h}\); \(c_{i}^{h}\) represents the feature generated by combining the \(i\)th word and the \(h\) words following it.  
\[c_{i}^{h}=f(w_{h}v_{i:i+h-1}+b), \tag{4}\]  
where \(f\) is a non-linear function, \(b\) is a bias, and:  
\[v_{i:i+h-1}=v_{i}\oplus v_{i+1}\oplus...\oplus v_{i+h-1}, \tag{5}\]  
where \(\oplus\) is the concatenation operator. According to the filter size, there are four different types of filters, including size three, size four, size five, and size six filters. We considered a filter in order to generate a new feature. The larger the filter size, the richer the context of consideration. In our experiment, we applied multiple filters to multiple features. CNN is characterized by parallelism, and each filter is not related to each other, which improves the execution efficiency.  
According to Figure 4, LSTM processes one code segment at a time, and the loop allows for information to be passed from one step of the network to the next. This chain-like nature reveals that the recurrent neural networks are intimately related to sequences. They are the genetic architecture of the neural network to use for such data.  
In the application of extracting sequence features, RNN can obtain more comprehensive inter-sequence information than CNN. In theory, CNN can only consider consecutive words' characteristics, and RNN can consider the entire sentence. However, in the experiment, the more information stored, the longer the processing time. Even if LSTM has chosen to forget some of the information, there is still the problem of prolonged time consumption for long sequences.