Within‐project and cross‐project just‐in‐time defect prediction based on denoising autoencoder and convolutional neural network

IET SoftwareVolume 14, Issue 3 p. 185-195 ArticleOpen Access Within-project and cross-project just-in-time defect prediction based on denoising autoencoder and convolutional neural network Kun Zhu, Kun Zhu School of Computer Science, Wuhan University, No.299 Bayi Road, Wuhan, People's Republic of ChinaSearch for more papers by this authorNana Zhang, Nana Zhang School of Computer Science, Wuhan University, No.299 Bayi Road, Wuhan, People's Republic of ChinaSearch for more papers by this authorShi Ying, Corresponding Author Shi Ying yingshl@whu.edu.cn School of Computer Science, Wuhan University, No.299 Bayi Road, Wuhan, People's Republic of ChinaSearch for more papers by this authorDandan Zhu, Dandan Zhu Artificial Intelligence Institute, Shanghai Jiaotong University, No.800 Dongchuan Road, Shanghai, People's Republic of ChinaSearch for more papers by this author Kun Zhu, Kun Zhu School of Computer Science, Wuhan University, No.299 Bayi Road, Wuhan, People's Republic of ChinaSearch for more papers by this authorNana Zhang, Nana Zhang School of Computer Science, Wuhan University, No.299 Bayi Road, Wuhan, People's Republic of ChinaSearch for more papers by this authorShi Ying, Corresponding Author Shi Ying yingshl@whu.edu.cn School of Computer Science, Wuhan University, No.299 Bayi Road, Wuhan, People's Republic of ChinaSearch for more papers by this authorDandan Zhu, Dandan Zhu Artificial Intelligence Institute, Shanghai Jiaotong University, No.800 Dongchuan Road, Shanghai, People's Republic of ChinaSearch for more papers by this author First published: 01 June 2020 https://doi.org/10.1049/iet-sen.2019.0278Citations: 17AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Just-in-time defect prediction is an important and useful branch in software defect prediction. At present, deep learning is a research hotspot in the field of artificial intelligence, which can combine basic defect features into deep semantic features and make up for the shortcomings of machine learning algorithms. However, the mainstream deep learning techniques have not been applied yet in just-in-time defect prediction. Therefore, the authors propose a novel just-in-time defect prediction model named DAECNN-JDP based on denoising autoencoder and convolutional neural network in this study, which has three main advantages: (i) Different weights for the position vector of each dimension feature are set, which can be automatically trained by adaptive trainable vector. (ii) Through the training of denoising autoencoder, the input features that are not contaminated by noise can be obtained, thus learning more robust feature representation. (iii) The authors leverage a powerful representation-learning technique, convolution neural network, to construct the basic change features into the abstract deep semantic features. To evaluate the performance of the DAECNN-JDP model, they conduct extensive within-project and cross-project defect prediction experiments on six large open source projects. The experimental results demonstrate that the superiority of DAECNN-JDP on five evaluation metrics. 1 Introduction In recent years, people are increasingly interested in just-in-time (JIT) defect prediction (i.e., change-level defect prediction) because it enables developers to identify defect-inducing changes within a specified inspection time, and the number of codes that needs to be reviewed is less (only need to review a single change, not the entire module) [1]. Defect-inducing change is a software change that introduces one or more defects into the source code in the software system (i.e. single or multiple consecutive commits within a given time period) [2]. JIT defect prediction may divide into within-project defect prediction (WPDP) and cross-project defect prediction (CPDP). Moreover, JIT defect prediction is a fine-grained defect prediction compared to traditional defect prediction at the module (such as package, file and class) level [3]. Currently, JIT defect prediction mainly uses traditional machine learning algorithms. However, traditional machine learning algorithms have their own limitations. We cannot achieve good results by using traditional machine learning algorithms for JIT defect prediction. For example, Kamei et al. [4] proposed the classic model of JIT defect prediction technology – EALR, which takes advantage of the following advantages of logistic regression (LR): (i) LR is suitable for scenes that need to obtain the classification probability, and JIT defect prediction also requires the classification probability. (ii) In terms of time and memory, the calculation cost of LR is not expensive, and large amounts of data can be processed with less resources. However, LR has three obvious shortcomings that severely constrain the performance of JIT defect prediction. First, the capability value size of each feature for the LR is calculated independently, in other words, LR cannot combine different features to generate new features. For example, given two features and , if is more important and is really a needed feature, inputting only and are not sufficient because LR cannot generate new feature . Second, LR can only achieve good effect when input features and the output label are in a linear relationship. Otherwise, the training effect will be poor or the training will fail. Third, the performance of LR is not very good when the feature space is large, and it is easy to underfit. Due to these three shortcomings of LR, the choice of input features will be very important. At present, deep learning is a research hotspot in the field of artificial intelligence, and has been successfully applied in many fields, such as speech recognition [5], image classification [6], natural language processing [7] and semantic search [8]. However, mainstream deep neural networks such as CNN and RNN have not been applied to improve the performance of JIT defect prediction. In this paper, in order to fill the research gap and consider the superior prediction performance of convolutional neural network (CNN) at the same time, we propose the denoising autoencoder CNN JIT defect prediction (DAECNN-JDP) model based on CNN to improve the performance of JIT defect prediction. The biggest advantage of CNN over LR is that it can construct basic change features into abstract deep semantic features. The deep semantic features may include , and even more complex non-linear combination of basic features that may be more capable of expressing the nature of the problem. If we use the abstract deep semantic features instead of basic features for JIT defect prediction, above these shortcomings of LR can be overcome. In addition, we leverage denoising autoencoder (DAE) [9] to enhance the robustness of basic change features, so as to improve the anti-interference ability of the DAECNN-JDP model. When basic change features have a certain degree of disturbance, the output features remain unchanged. The DAE removes noise through training, and obtains the features that are not contaminated by noise, thereby the learned feature expression has the stronger robustness and generalisation ability. The main contributions of this paper are as follows: We propose a novel JIT defect prediction model named DAECNN-JDP, which leverages the CNN to combine basic defect features into abstract deep semantic features. We set different weights for the position vector of each dimension feature, which can be automatically trained by adaptive trainable vector according to the features of defect instances. We also utilise the DAE to learn the more robust change feature representation, so as to improve the anti-interference ability of the DAECNN-JDP model. We conduct extensive within-project and cross-project JIT defect prediction experiments on six large open source projects, and compare our DAECNN-JDP model with 11 baseline models, including eight machine learning models, EALR, Deeper and CNN-JDP. The experimental results show that the DAECNN-JDP model outperforms these baseline models in terms of PofB20, accuracy, precision, recall and F1. The rest of our paper is organised as follows. Section 2 describes the background and related work. Section 3 details our proposed DAECNN-JDP model. Section 4 shows the experimental setup, including data sets, evaluation metrics and baseline models. Section 5 evaluates the performance of our DAECNN-JDP model. Section 6 analyses the advantages of our DAECNN-JDP model and discusses the selection scheme of the hyperparameter in practice. Section 7 introduces the threats to validity. We conclude the paper and describe the future work in Section 8. 2 Background and related work In this section, we introduce the background and related work from the following three parts: JIT defect prediction, DAE and the application of deep learning techniques in software engineering. 2.1 JIT defect prediction Mockus and Weiss [10] first proposed a prediction model that focuses on identifying the defect-inducing software change in 2000. Such change level prediction model is also referred to as JIT defect prediction by Kamei et al. in 2013 [4]. JIT defect prediction may divide into WPDP and CPDP. JIT defect prediction is more practical because (i) JIT defect prediction is built on the level of code change, and the number of codes that needs to be reviewed is less and (ii) developers can review and test these risky changes while their minds are still clear (for example, at the time of submission), and they can perform JIT defect prediction before the developer submits the current changes. Therefore, JIT defect prediction has become a more practical method for practitioners to carry out development work. Kamei et al. [4] conducted a large-scale empirical study of six open source projects and five commercial projects. They used a modified linear regression model (EALR) to detect 20% of the lines of code, and the experimental results show that 35% of the defect-inducing changes are detected. This work is very important because it can identify more defect-inducing changes for each unit's code inspection work. Yang et al. [11] propose a JIT defect prediction model named Deeper, which builds initial change features into expressive features through deep belief network (DBN) algorithm, and then uses the new expressive features to establish the defect prediction model. They conduct experiments on six large open source projects, and the results show that the Deeper model can discover 51.04% of defects on average and achieve F1 of 0.22–0.63. Fukushima et al. [12] introduced a cross-project model of JIT defect prediction. The results show that it is feasible to use data from other projects to conduct JIT defect prediction. Shihab et al. [13] conduct an industrial study on the risky software changes. They find that the number of lines of code added, the complexity of file change, the number of bug reports associated with changes, and the developer's experience are the best indicators of measuring risky change. In the study of Hoang et al. [14], the DeepJIT model only uses simple CNN to extract features from commit messages and code changes, and to conduct an end-to-end training (there is no output in the middle process of CNN), so as to use these trained features to identify defects. Similar to their study, we also use the CNN technique, and the structure of CNN also includes the input layer, convolutional layer, relu layer, full connection layer, output layer and so on. Compared with the study of Hoang and co-workers, in addition to using CNN, our DAECNN-JDP model also use feature weight training based on position and DAE techniques, and integrate the squared error loss function of DAE with the sparse cross entropy loss function of CNN by controlling and adjusting hyperparameter . 2.2 Denoising autoencoder The autoencoder is a kind of neural network which aims at inputting the reconstruction signal. It has a hidden layer h, which can generate encoding to represent the input, and map the input x to the output r by decoding h. The autoencoder consists of two parts: an encoder represented by the function h = f(x) and a decoder r = g(h) that generates the reconstruction. The features generated by the autoencoder are often used to replace the original features to achieve better effect in deep learning. In software defect prediction, it is not enough to achieve good features only by reconstructing the input feature. It is also necessary to make these features have anti-interference ability, i.e. when the input feature has a certain degree of disturbance, the generated features remain unchanged. In this case, it is possible to add more difficulty to the model by adding noise, and the trained model will have better robustness. Therefore, Vincent et al. [9] propose an improved version of the autoencoder DAE. The DAE adds noise to the training features based on the autoencoder, and the output features are still the original samples (without adding noise), so the denoising encoder must learn to remove the noise and obtain the real input features that are not contaminated by noise. Therefore, this forces the denoising encoder to learn the more robust feature representation of the input features and the obtained features have more stronger generalisation capability. At present, DAE has been applied to many aspects. Li et al. [15] use the mapping weight of DAE to initialise neural network. After fine-tuning the weight, the trained neural network is used to identify the classes of motor imageries. Ferles et al. [16] propose a self-organising map model (DASOM) for DAE, which integrates DAEs into a hierarchically organised hybrid model. Vincent [17] proved the rationality of using binding weights between the encoder and the decoder, and successfully extend DAE to a larger series of energy models. Wang et al. [18] use DAE to integrate the ratings and the explicit trust relationships between users in the social network in order to simulate the user's preferences more accurately. 2.3 Application of deep learning techniques in software engineering Recently, some researchers adopt deep learning algorithms to improve tasks in software engineering. Wang et al. [19] leverage DBN to extract token vectors from the program's abstract syntax tree to automatically learn semantic features. Their evaluation results show that the semantic features significantly improve WPDP and CPDP in ten open source projects compared to traditional features. Lam et al. [20] combine deep learning with information retrieval to improve the performance of locating defect files. Huo et al. [21] use CNN to learn the features from the source code in program and the natural language in the defect report, and then combine two kinds of features as the unified feature of defect location. Gu et al. [22] leverage the RNN encoder–decoder model to solve the problem of retrieving API call sequences based on the user's natural language query. Deep learning is also used for test report classification [23], link prediction in developer online forums [24], software traceability [25] and so on. 3 Methodology In this section, we will introduce the details of the DAECNN-JDP model. We first set different weights for the position vector of each dimension feature, which can be automatically trained by adaptive trainable vector according to the features of defect instances. Then we obtain the input features that are not contaminated by noise through the training of DAE, so as to learn more robust feature representation. Finally, we construct the basic change features into the abstract deep semantic features by the CNN, so as to better predict whether the change of unknown label is really defective or not. Fig. 1 describes the detailed workflow of the DAECNN-JDP model, which includes five steps from (a)–(e). The model consists of two stages: the model construction stage and the defect prediction stage. In the model construction stage, we leverage the CNN to train the change feature of known label (i.e. defective or non-defective), so as to construct the CNN classifier. In the defect prediction stage, we leverage the CNN classifier constructed in the model construction stage to predict whether the change of the unknown label is defective or not. Fig. 1Open in figure viewerPowerPoint Framework of the DAECNN-JDP model (a) Feature extraction, (b) Feature preprocessing, (c) Weight training based on position, (d) Feature presentation based on DAE, (e) Feature integration, training and prediction based on CNN In short, in the model construction stage, the DAECNN-JDP model mainly consists of five steps: (a) feature extraction, (b) feature preprocessing (class imbalance processing and data normalisation), (c) feature weight training based on position, (d) robustness feature expression based on DAE, (e) feature integration and training based on CNN. In the defect prediction stage, the DAECNN-JDP model does not contain the step (d) robustness feature expression based on DAE, which includes only four steps: (a) feature extraction, (b) feature preprocessing (we use the change feature of the unknown label in the prediction stage, so only need to perform data normalisation and does not need to perform the class unbalance processing), (c) feature weight training based on position, (e) feature integration and prediction based on CNN. 3.1 Feature extraction We first extract multiple quantifiable change features from the history change data, which can distinguish whether the change is defective or not. We divide the extracted change instances into change instances for training and change instances for prediction. In this paper, we use 14 basic defect change features proposed by Kamei et al. [4], as shown in Table 1. Table 1. Name and description of 14 basic change features Name Description NS the number of modified subsystems [10] ND the number of modified directories [10] NF the number of modified files [26] entropy distribution of modified code across each file [27] LA lines of code added [28] LD lines of code deleted [28] LT lines of code in a file before the change [29] FIX whether or not the change is a defect fix [30] NDEV the number of developers that changed the modified files [30] AGE the average time interval between the last and the current change [31] NUC the number of unique changes to the modified files [27] EXP developer experience [10] REXP recent developer experience [10] SEXP developer experience on a subsystem [10] Each data set contains 14 change features, which can be classified into the following five dimensions: diffusion, size, purpose, history and experience. The diffusion dimension consists of NS, ND, NF and Entropy, which can characterise the distribution of a change. A highly distributed change is more likely to be a defect-inducing change [4]. The size dimension contains LA, LD and LT, which characterises the size of a change. A larger change is expected to have a higher likelihood of being a defect-inducing change [32]. The purpose dimension only contains FIX. A defect-fixing change is more likely to introduce a new defect [33]. The history dimension contains NDEV, AGE and NUC. A defect is more likely to be introduced by a change if the touched files have been modified by more developers, or by more recent changes [27]. The experience dimension contains EXP, REXP and SEXP. If a change is made by a more experienced developer, it is less likely that a defect will be introduced [4]. Based on the above considerations, we complete the feature extraction by dividing the features into the above five dimensions. 3.2 Feature preprocessing Feature preprocessing in this paper includes two parts: class imbalance processing and data standardisation. In the model construction stage, we need to perform class imbalance processing and data standardisation; in the defect prediction stage, since we use the change instances of unknown label, so only need to perform data standardisation. 3.2.1 Class imbalance processing Class imbalance is a common problem in software defect data sets. The distribution of software defects in a project is roughly in line with the pareto principle, i.e. 20% of the program modules contain about 80% of the defects. For the defect prediction data set, the number of defective modules (a few class) is far less than that of the non-defective modules (majority class). If there is the serious class imbalance problem in the software defect data set, the prediction effect of the model is relatively poor. In this paper, we adopt the synthetic minority oversampling technique (SMOTE) method to perform the class imbalance processing [34]. This method is an improved scheme based on random oversampling algorithm. Since random oversampling adopts the strategy of simply copying samples to increase the number of a few class samples, the model is easy to produce the overfitting, and the SMOTE algorithm can avoid the problem of overfitting to a certain extent. The basic idea of SMOTE algorithm is to analyse a few class samples and manually synthesise new samples based on a few class samples, and add them to the currently existing data set at the same time. In summary, the SMOTE algorithm synthesises new samples for a few classes based on 'interpolation'. This step is critical for JIT defect prediction, because it can help the trained classifier does not bias towards non-defective modules (majority class), thereby improving the performance of JIT defect prediction. 3.2.2 Data standardisation The distribution of the 14 basic defect change feature values extracted is of large difference, even not in the same order of magnitude. If the original feature values are used for JIT defect prediction, the function of the higher values in the comprehensive analysis will be highlighted, and the function of the lower values is relatively weakened. Therefore, in order to ensure the reliability of the result, we need to standardise the original feature values, and require the processed result to be consistent with feature values of the original distribution. In this paper, we use the z-score standardisation method [3], for each value of the feature f, the normalised value is calculated as follows: (1)where and are the mean and variance of the initial values in feature f, respectively. 3.3 Feature weight training based on position The six data sets used in this paper all have 14-dimensional features. Each dimensional feature has different effect on JIT defect prediction, i.e. the capability value of each dimensional feature is different. In order to highlight important features and weaken unimportant ones, we set different weights for the position vector () of each dimensional feature, which can be automatically trained by adaptive trainable vector () according to the features of defect instances. As shown in Fig. 2, we establish a channel for each dimensional feature, and the weight of each channel is controlled between [0,1]. Where 0 means that the features of the channel position do not play any role, and the closer the weight value is to 1, the more important the feature is. Fig. 2Open in figure viewerPowerPoint Feature weight representation schematic diagram based on position In our method, for the following (2), is a control gate weight vector automatically trained by an adaptive trainable function, and then the vector can be transformed by a Sigmoid nonlinear activation function to obtain the final weighted vector Sigmoid(), which can assign different weights to each dimensional feature, thus distinguishing the importance degree of different features. (2)where , . 3.4 Robustness feature representation based on DAE In the model construction stage, we use DAE to further process these software defect features. DAE can remove noise through training to obtain true input features that are not contaminated by noise. The defect data processed by the DAE has more robust feature representation and stronger generalisation capability. Then, we put the robust features obtained through the DAE processing into the CNN for training, which can achieve better classification effect. In the defect prediction stage, there is no need to use DAE. DAE takes the corrupted data as the input and the predicted raw undamaged data as the output. It can learn useful information by changing the reconstruction error term. Since it is trained on the corrupted data, it can also be used to repair the lost data. The training process of DAE is shown in Fig. 3. DAE is trained to reconstruct clean data points x from damaged version . This can be achieved by minimising the loss , where is the damaged version of the instance x after the damage process . DAE can learn the reconstruction distribution from the training data pairs according to the following procedure: Collect the training instances x from the training set. Fig. 3Open in figure viewerPowerPoint Training process of DAE Pick the damaged instances from . We introduce a damage process , the conditional distribution represents the probability that the given data instances X produce the corrupted instances . We add Gaussian noise in the software defect prediction data sets. Gaussian noise is a type of noise whose probability density function obeys the Gaussian distribution (i.e., normal distribution). The probability density function is as follows: (3)where is the expectation and is the standard deviation. We mix the input data with the random number in accordance with Gaussian distribution to obtain Gaussian noise data . iii) Use as the training instance to estimate the reconstructed distribution of the DAE, where h is the output of encoder , is defined according to the decoding function . The d-dimensional input data is mapped to the dimensional hidden layer h, and then reconstruct to obtain the d-dimensional output r. The mathematical equations for these two steps are as follows: (4) (5)where represents Gaussian noise data, W represents the ()dimensional weight from the input layer to the hidden layer, is the hidden layer bias, and s represents the activation function, so the parameter of the encoding layer is ; represents the () dimension weight from the hidden layer to the reconstruction layer, is the output layer bias, so the parameter of the decoding layer is . The equation of is as follows: (6)where L is the squared error loss function. Our DAE is a feedforward network, which performs random gradient decline under the expectation of (6) (7)where is the distribution of the training data. The loss function of the DAE uses the squared error (the average reconstruction error), which is the square of the difference between the prediction value and the true value. The smaller the value, the better the performance of DAE model is. The loss function Loss1 of DAE is as follows: (8)where is the norm of the squared error. 3.5 Feature integration, training and prediction based on CNN In the model construction stage, we construct the basic change features of known label into the abstract deep semantic features through CNN. The CNN trained by the deep semantic features has stronger classification capability. In the defect prediction stage, we leverage the trained convolution neural network to predict whether the change of unknown label is defective or non-defective. The training of CNN is mainly divided into backward calculation and forward calculation. For the backward calculation (back propagation), the local gradient of neurons in each layer is as follows: (9)For the forward calculation, we update the connection weights between the neurons in each layer and the neurons in the previous layer. The Delta rule is as follows: (10)In the convolution neural network section, the structure of CNN is shown in Fig. 4. We map 14-dimension features outputted from DAE to 16-dimension features, and then reshape into a 4 × 4 matrix, so that these features can be better for convolutional processing. We perform the three-layer convolution process on the matrix. The number of neurons in each convolutional layer is 32, 64 and 16, respectively, and the convolution kernel sizes are 3 × 3, 3 × 3, and 4 × 4, respectively. We add a Relu non-linear activation function layer behind each convolutional layer. After completing the convolution operation, we reshape the matrix back to 16-dimensional features, and then we tile the 16-dimension features through a log fully connected layer. Finally, we use the softmax layer to classify the test set of the software defect data, and the test set is divided into defective changes or non-defective changes, so as to complete the JIT defect prediction. Fig. 4Open in figure viewerPowerPoint Structure of CNN The loss function of the CNN uses sparse cross entropy. In terms of classification, it is the probability that the input samples are predicted to belong to a certain class. The smaller the cross entropy is, the more accurate the prediction result will be. The equation for sparse cross entropy is as follows: (11)where t represents the true value classification (0 or 1) and a represents th