# An Improved CNN Model for Within-Project Software Defect Prediction
## 2 Background
### Convolutional Neural Network  
A CNN is a special type of neural network used to process data that has a known, grid-like topology [26]. Examples include one-dimensional (1D) time-series data as well as two-dimensional (2D) and three-dimensional (3D) image data. CNN has been tremendously successful in practical applications, including speech recognition [32], image classification [25], and natural language processing [33; 34]. In this work, we leveraged a CNN model to extract features from AST programs automatically.  
Figure 2 demonstrates the overall architecture of a CNN. A CNN consists of convolutional layers and pooling layers. In a simple fully-connected network, which we call dense network, neuron units are connected to all neuron units of its neighboring layers. In CNN, neural units connected to these two layers are sparsely connected, which is determined by kernel size and pooling size. A CNN network features two key characteristics that are sparse connectivity and shared weights. These two characteristics help reduce model capacity and capture global patterns rather than local patterns.  
Figure 1: A typical software defect prediction process based on machine learning.  
Figure 2: A basic convolutional neural network (CNN) architecture.  
Sparse connectivity means that each neuron is connected to only a limited number of other neurons [35]. In a CNN, sparse connectivity is controlled by kernel size and pooling size. Let us take node V\({}_{3}\) in Figure 2 as an example. When kernel size = 3, it is connected to only three nodes h\({}_{1}\), h\({}_{2}\), and h\({}_{3}\), whereas, h\({}_{4}\) is not affected by V\({}_{3}\). In the same way, h\({}_{2}\) is affected only by V\({}_{2}\), V\({}_{3}\), and V\({}_{4}\). Each subset connecting to the next layer in the CNN is called a local filter, which captures a specific kind of patterns. To calculate the output to the next layer, one can multiply each local filter by outputs from the previous layer, add a bias, and then perform a nonlinear transformation. In Figure 2, if we denote the ith neuron in the mth layer (convolutional layer) as \(h_{i}^{m}\), the weights of the ith neuron in the \((m-1)\)th layer as \(W_{i}^{m-1}\), the bias in the \((m-1)\)th layer as \(b^{m-1}\), and we use ReLU as our activation function, then the output can be calculated as follows:  
\[h_{i}^{m}=ReLU\big{(}W_{i}^{m-1}*V_{i}^{m-1}+b^{m-1}\big{)}. \tag{1}\]  
Shared weight means that each filter shares the same parameterization (weight vector and bias) [24]. For example, in Figure 2 all the solid black lines linking the input layer and convolutional layer share the same parameters, and the same is true of the blue sparse-dotted lines and the orange dense-dotted lines. Shared weight enables a CNN to capture features independent of their positions and it can also effectively reduce model capacity.  
In CNN, a pooling layer is usually used after a convolutional layer. It replaces the output of the network at a certain location with a summary statistic of the nearby outputs [35]. Pooling helps to make representations more robust, in other words, approximately invariant to small changes in the input. Moreover, it can reduce the numbers of intermediate representations which translates into reduced model capacity.  
There are many hyperparameters in CNN models including filter size, pooling size, and convolutional and pooling layer numbers. These hyperparameters must be tuned to make the model work well. We will discuss the results of our hyperparameter tuning process in Section 5.3.