Software defect prediction based on stacked sparse denoising autoencoders and enhanced extreme learning machine

IET SoftwareVolume 16, Issue 1 p. 29-47 ORIGINAL RESEARCH PAPEROpen Access Software defect prediction based on stacked sparse denoising autoencoders and enhanced extreme learning machine Nana Zhang, Nana Zhang orcid.org/0000-0002-5256-9245 School of Computer Science, Wuhan University, Wuhan, ChinaSearch for more papers by this authorShi Ying, Corresponding Author Shi Ying yingshi@whu.edu.cn whu01y@163.com orcid.org/0000-0003-4291-5586 School of Computer Science, Wuhan University, Wuhan, China Correspondence Shi Ying, School of Computer Science, Wuhan University, No.299 Bayi Road, Wuhan, China. Email: yingshi@whu.edu.cn and whu01y@163.comSearch for more papers by this authorKun Zhu, Kun Zhu School of Computer Science, Wuhan University, Wuhan, ChinaSearch for more papers by this authorDandan Zhu, Dandan Zhu Artificial Intelligence Institute, Shanghai Jiaotong University, Shanghai, ChinaSearch for more papers by this author Nana Zhang, Nana Zhang orcid.org/0000-0002-5256-9245 School of Computer Science, Wuhan University, Wuhan, ChinaSearch for more papers by this authorShi Ying, Corresponding Author Shi Ying yingshi@whu.edu.cn whu01y@163.com orcid.org/0000-0003-4291-5586 School of Computer Science, Wuhan University, Wuhan, China Correspondence Shi Ying, School of Computer Science, Wuhan University, No.299 Bayi Road, Wuhan, China. Email: yingshi@whu.edu.cn and whu01y@163.comSearch for more papers by this authorKun Zhu, Kun Zhu School of Computer Science, Wuhan University, Wuhan, ChinaSearch for more papers by this authorDandan Zhu, Dandan Zhu Artificial Intelligence Institute, Shanghai Jiaotong University, Shanghai, ChinaSearch for more papers by this author First published: 31 May 2021 https://doi.org/10.1049/sfw2.12029AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Software defect prediction is an important software quality assurance technique. Nevertheless, the prediction performance of the constructed model is easily susceptible to irrelevant or redundant features in the software projects and is not predominant enough. To address these two issues, a novel defect prediction model called SSEPG based on Stacked Sparse Denoising AutoEncoders (SSDAE) and Extreme Learning Maching (ELM) optimised by Particle Swarm Optimisation (PSO) and another complementary Gravitational Search Algorithm (GSA) are proposed in this paper, which has two main merits: (1) employ a novel deep neural network – SSDAE to extract new combined features, which can effectively learn the robust deep semantic feature representation. (2) integrate strong exploitation capacity of PSO with strong exploration capability of GSA to optimise the input weights and hidden layer biases of ELM, and utilise the superior discriminability of the enhanced ELM to predict the defective modules. The SSDAE is compared with eleven state-of-the-art feature extraction methods in effect and efficiency, and the SSEPG model is compared with multiple baseline models that contain five classic defect predictors and three variants across 24 software defect projects. The experimental results exhibit the superiority of the SSDAE and the SSEPG on six evaluation metrics. 1 INTRODUCTION With the growth of software scale and complexity, the number of defects embedded in software increases dramatically, which further increases software development and maintenance costs and reduces customer satisfaction [1]. To solve this issue, software defect prediction (SDP) techniques have been proposed to effectively guide the direction of software testing by reasonably allocating limited development and maintenance resources [2]. In the widely used defect prediction datasets, each defect instance is composed of a series of software features and a class label. Jiarpakdee et al. [3] have demonstrated that 10%-67% of features in the 101 open source defect datasets are irrelevant or redundant. If all these software features are used to construct the defect prediction model, the prediction performance and generalisation capability of the model will be severely reduced. Consequently, it is very necessary to conduct feature selection or extraction for defect datasets in software defect prediction. Previous studies [4, 5] have shown that the prediction performance of the model can significantly boost when applying feature selection or extraction techniques to the defect datasets. Feature selection techniques reduce the number of features in a model by selecting an optimal representative feature subset, while feature extraction techniques decrease the number of features by constructing new, combined features from the original features. Currently, most studies mainly adopt feature selection techniques for defect prediction, feature extraction techniques have not been thoroughly investigated in software defect prediction. For feature extraction techniques, most researchers use Principal Component Analysis (PCA) [6] and Kernel Principal Component Analysis (KPCA) [7] to remove irrelevant and redundant features, and these traditional methods cannot well represent inherent structure hidden behind the original defect features. At present, deep learning techniques have been successfully applied in many fields by constructing a deep network architecture to automatically learn deep semantic feature representation, such as speech recognition [8], image classification [9, 10]. Previous studies have shown that deep semantic features have stronger discriminating capacity for different classes (defective or non-defective) [11, 12]. For the above reasons, we leverage a novel deep neural network – Stacked Sparse Denoising AutoEencoders (SSDAE) [13] to combine traditional features into new features, which can effectively learn the robust deep semantic feature representation. Not only the sparse constraint is introduced to compress the input data, but also the feature extraction capability is further enhanced by stacking multiple sparse denoising autoencoders. In this paper, we adopt an advanced machine learning method – Extreme Learning Machine (ELM) [14] as the defect predictor for software defect prediction based on the deep semantic features extracted by SSDAE. The ELM has obvious advantages in classification, including strong classification capacity, fast training speed and good generalisation performance [15]. Nevertheless, the initial input weights and hidden layer biases of the ELM are set randomly. The selection of these weights and biases has a great impact on the prediction performance, and we cannot obtain the optimal weights and biases. For this shortcoming of ELM, we try to utilise metaheuristic intelligence optimisation algorithms to determine the optimal input weights and hidden layer biases of ELM, including Gravitational Search Algorithm (GSA) [16] and Particle Swarm Optimisation (PSO) [17]. Among them, GSA [16] was proposed by Rashedi et al. as a new metaheuristic physics-based algorithm, which uses physical rules to find the best solution in the problem search space. This algorithm has the advantages of robustness, effortlessness and strong exploration capacity. We can leverage strong exploration capacity of GSA to optimise the input weights and hidden layer biases of ELM. The GSA is capable of quickly finding the global optimal solution of the parameter combination by speeding up the convergence rate based on the initial parameters adjusted properly. Unfortunately, as the generation increases, the convergence rate that the GSA obtains the global optimal solution gradually slows down, and the GSA is easy to fall into the local optimal solution [18]. For this reason, PSO is used in combination with GSA to effectively overcome the deficiency of GSA in this paper. The PSO algorithm has been widely applied and popularised due to its strong exploitation capacity (fast convergence property) and simple evolutionary operators. The hybrid PSO-GSA algorithm can incorporate strong exploitation capacity (fast convergence property) of PSO into strong exploration capability of GSA. Li et al. [19] have verified the validity of this optimisation model in some regression and classification benchmark problems. We regard the input weights and hidden layer biases of the ELM as the agent of the hybrid PSO-GSA algorithm, and choose the matrix encoding strategy to represent each agent. The main contributions of this paper are as follows: (1) Motivated by the predominant performance of deep learning technique and the idea of search-based software engineering, we propose a novel defect prediction model called SSEPG based on SSDAE and ELM optimised by PSO and another complementary GSA in this paper. (2) We employ a novel deep neural network – Stacked Sparse Denoising AutoEncoders (SSDAE) to extract new combined features, which can effectively learn the robust deep semantic feature representation. On the one hand, the sparsity constraint is introduced to avoid overfitting and improve generalisation capability; on the other hand, multiple sparse denoising autoencoders are stacked to further enhance the features extraction capability. (3) We integrate strong exploitation capacity of PSO with strong exploration capability of GSA to optimise the input weights and hidden layer biases of ELM, and utilise the superior discriminability of the enhanced ELM to predict the defective modules. (4) We conduct extensive experiments for feature extraction and defect prediction across 24 software defect projects from large open source datasets. We compare the SSDAE with eleven state-of-the-art feature extraction methods in effect and efficiency, and compare the SSEPG model with multiple baseline models that contain five classic defect predictors and three variants. The experimental results demonstrate that the SSDAE and the SSEPG can significantly boost the prediction performance on six evaluation metrics basically. To validate the performance of the SSDAE and the SSEPG, we set the following seven research questions (RQs). In RQ1, we investigate the effect of features extracted by SSDAE on model performance compared with eleven state-of-the-art feature extraction methods. We also compare SSDAE with these eleven state-of-the-art feature extraction methods in terms of time cost in RQ2. For RQ3, we explore the prediction performance of SSEPG and five classic defect predictors with the same feature extraction method SSDAE. Similarly, we examine the prediction performance of SSEPG and its three variants in RQ4. In QR5, we verify the effect of deep semantic features extracted by SSDAE compared with the traditional features (without SSDAE) on model performance. In RQ6, we investigate whether our SSEPG model has performance differences on the same software system with different releases. We also explore how different defect ratios in a software project affect model performance in RQ7. The reminder of this paper is organised as follows. Section 2 describes the related work. Section 3 details our proposed SSEPG model. Section 4 shows the experimental setup, including benchmark datasets, evaluation metrics, baseline methods and Scott-Knott Effect Size Difference (ESD) test. Section 5 evaluates the performance of our SSEPG model. Section 6 discuss the possible reasons that affect the model performance. Section 7 introduces the threats to validity. We conclude this paper and describe the future work in Section 8. 2 RELATED WORK In this section, we briefly introduce the typical software defect prediction models, feature selection and extraction methods in software defect prediction. 2.1 Software defect prediction Many researchers have proposed various models for predicting whether a module is defective, including supervised models, semi-supervised models, and unsupervised models. Supervised models include support vector machines [20, 21], neural networks [22], Bayesian models [23, 24], decision trees [25, 26] and transfer learning [27]. However, the number of training data with labels may be insufficient. Some researchers construct semi-supervised models to predict software defect modules. For instance, Ma et al. [28] proposed an improved semi-supervised learning model for solving the problems of class imbalance and data insufficient with labels in software defect prediction. Lu et al. [29] proposed an iterative semi-supervised approach by repeatedly fitting the fixed status of unlabelled data. For unsupervised defect prediction, some researchers have tried to address the problem of data insufficient with labels by adopting unsupervised learning. Nam et al. [30] proposed an unsupervised defect prediction model named CLAMI on unlabelled datasets by using an automated manner. In addition, some researchers conducted large-scale empirical studies to compare the performance differences of various models. Elish et al. [31] compared support vector machine with other eight machine learning algorithms on the NASA data repository. The experimental results showed that support vector machine is superior to other eight methods. Different from the previous studies that only use machine learning methods for software defect prediction, we try to utilise the machine learning method (ELM) optimised by a hybrid metaheuristic intelligence PSO-GSA algorithm to boost the prediction performance in this paper. 2.2 Feature selection and extraction methods in software defect prediction Previous studies have shown that feature selection or feature extraction techniques can improve the prediction performance of the model [4, 5]. Feature selection techniques reduce the number of features in a model by selecting an optimal representative feature subset, while feature extraction techniques decrease the number of features by constructing new, combined features from the original features. Currently, most studies mainly use feature selection techniques for defect prediction; feature extraction techniques have not been thoroughly investigated in software defect prediction. Most feature selection techniques belong to filter-based feature ranking methods or wrapper-based feature subset selection methods [32]. For filter-based feature ranking methods, the feature space will be searched based on the data itself rather than a specific learning algorithm, such as information gain and chi-square statistics [33, 34]. Wrapper-based feature subset selection methods need to use a specific learning algorithm (classification algorithm or regression algorithm) to evaluate the performance of various feature subsets in the selection process [35, 36]. In addition, some intelligence optimisation algorithms have been widely used in feature selection. Kashef et al. [37] proposed an advanced binary ACO algorithm for feature subset selection, and the experimental results showed that this algorithm can achieve better performance with a smaller feature subset. Xiong et al. [38] proposed a hybrid feature selection algorithm that introduces dynamic weighted technique into the ACO algorithm. Chuang et al. [39] leveraged chaotic binary particle swarm optimisation for the feature selection. Sarafrazi et al. [40] proposed a hybrid GSA-SVM algorithm by extracting the appropriate feature subset in binary problems. For feature extraction techniques, most researchers use principal component analysis (PCA) [6] and kernel principal component analysis (KPCA) [7] to remove irrelevant and redundant features. Different from the previous studies, we leverage a novel deep neural network – Stacked Sparse Denoising AutoEncoders (SSDAE) to learn the robust deep semantic feature representation. 3 METHODOLOGY In this paper, we propose a novel software defect prediction model called SSEPG, which consists of two stages: (1) Feature extraction based on SSDAE. We employ an advanced deep neural network – Stacked Sparse Denoising AutoEncoders (SSDAE) to combine original features into new features, which can effectively learn the robust deep semantic feature representation. Not only the sparse constraint is introduced to compress the input data, but also the feature extraction capability is further enhanced by stacking multiple sparse denoising autoencoders. (2) ELM based on the hybrid PSO-GSA optimisation. We leverage PSO and another complementary GSA to optimise the input weights and hidden layer biases of ELM. The hybrid PSO-GSA algorithm can incorporate strong exploitation capacity (fast convergence property) of PSO into strong exploration capability of GSA. We utilise the discriminating capacity of the enhanced ELM predictor to conduct defect prediction according to the deep semantic features extracted by SSDAE. 3.1 Feature extraction based on SSDAE SSDAE aims to reduce the number of features by constructing new, combined features from the original defect features, and extract the robust deep semantic features by a series of learning stages that minimise the reconstruction error. We first analyse the basic building component of SSDAE – stack denoising autoencodes, which attempts to reconstruct a clean 'repaired' input from the corrupted input. The advantages of using the stack denoising autoencodes are as follows: (1) The learning deep semantic feature representation is rather stable and robust under corrupted input. (2) The extracted features can capture the useful structure in the input distribution by the denoising process. The stack denoising autoencodes consists of a series of Denoising AutoEncodes (DAEs). For each DAE, given a d-dimensional input vector x ∈ R d , its corrupted version x ∼ can be obtained by introducing Gaussian noise. The Gaussian noise is a type of noise whose probability density function obeys the Gaussian distribution. The probability density function is as follows: f ( x ) = 1 2 π σ e x p ( − ( x − μ ) 2 2 σ 2 ) (1)where μ is the expectation and σ is the standard deviation. In addition to the damage process, DAE mainly consists of two processes: encoding and decoding. First, the corrupted version x ∼ with Gaussian noise is mapped to the k-dimensional hidden layer h, which is an encoding process, as shown in Equation (2); and then the hidden layer h is reconstructed to the d-dimensional output layer h ˆ , which is a decoding process, as shown in Equation (3). h = f ( W x ˜ + b 1 ) (2) h ˆ = f ˆ ( W ′ h + b 2 ) (3)where W ∈ R d × k and W ′ ∈ R k × d are the weight matrices of the encoder and decoder, respectively, b 1 ∈ R k and b 2 ∈ R d are the bias vectors of hidden layer and output layer, respectively, f(.) and f ˆ ( . ) represent the sigmoid activation functions, h ˆ is an approximation of the original input x. The parameter set θ = {W, W′, b1, b2} for DAE is trained to minimise the reconstruction error, as shown in Equation (4). L DAE θ = 1 N ∑ i = 1 N x i − h ˆ i 2 2 + μ 2 ( W F 2 + W ′ F 2 ) (4)where N is the total number of training instances, μ is a weight constraint term that can reduce the magnitude of the weight and prevent overfitting. After training the first DAE, we continue to train the next layer by considering the hidden layer of the first DAE as the input of the next DAE, and these DAEs are stacked to form a stack denoising autoencodes. In general, the more nodes in the hidden layer, the better the reconstructed input data. But when the number of nodes in the hidden layer far exceeds the number of nodes in the input layer, the Sparse AutoEncoder (SAE) is introduced to compress the input data. If the output of a hidden node is close to 1, we consider that it is activated, and when the output is close to 0, it is considered to be suppressed. The limitation that the hidden nodes are suppressed at most times is called sparsity constraint, which can make the average activation of hidden nodes extremely small. We adopt h(xj) as the activation of the jth hidden node and define the average activation state of jth hidden node as: χ ˆ j = 1 N ∑ i = 1 N h ( x j ) (5) The Kullback-Leibler (KL) divergence between χ and χ ˆ j is used to describe the constraint, as shown in Equation (6) K L ( χ ˆ j ‖ χ ) = χ log χ χ ˆ j + ( 1 − χ ) log 1 − χ 1 − χ ˆ j (6)where χ is a sparsity variable that denotes the sparsity target. The larger the KL divergence, the larger the difference between χ and χ ˆ j , and if the KL divergence equals to 0, it means that χ = χ ˆ j . To prevent overfitting and to combine the robustness of DAE and the sparsity of SAE, SAE and DAE are cascaded to form Sparse Denosing AutoEncoder (SDAE). We add an additional penalty factor – the sparsity constraint term to the reconstruction error of DAE, and train the DAE to minimise the loss regularised by this sparsity-inducing term. Hence the loss function of SDAE is as follows: L SDAE θ = 1 N ∑ i = 1 N x i − h ˆ i 2 2 + ν ∑ j = 1 s K L ( χ ˆ j ‖ χ ) + μ 2 ( W F 2 + ‖ W ′ ‖ F 2 ) (7)where s is the number of hidden nodes in the hidden layer, ν is a sparsity weight coefficient. SSDAE is a deep neural network in which multiple SDAEs are stacked on top of each other. Compared with SDAE, SSDAE has the following three advantages: (1) fit training samples better; (2) extract deep semantic feature representation; (3) enhance the learning capacity and generalisation capacity of the network. In this paper, SSDAE is composed of two SDAEs, and the structure of SSDAE is shown in Figure 1. SSDAE adopts the unsupervised layer-by-layer training, which can alleviate the gradient dispersion phenomenon to a certain extent. First, each SDAE is trained separately to obtain the optimal weights of the network, and then these weights are taken as the initial values of the SSDAE network weights. Finally, the back propagation (BP) algorithm is used for overall fine-tuning until the optimal parameters of the entire network are obtained. The hidden layer for the previous SDAE is used as the input of the subsequent SDAE. Since the pre-trained weights had been regularised to the network, we remove the sparsity constraint term here. The loss function of SSDAE in the fine-tuning phase is as follows: L SSDAE θ = 1 N ∑ i = 1 N x i − h ˆ i 2 2 + μ 2 ∑ r = 1 R ( W ( r ) F 2 + ‖ W ′ ( r ) ‖ F 2 ) (8)where r is the number of SDAE in SSDAE, and r is equal to two in this paper. FIGURE 1Open in figure viewerPowerPoint The structure of the Stacked Sparse Denoising AutoEncoders network After completing feature extraction by SSDAE, the original defect data x is transformed to a new dataset x' for each software project. 3.2 ELM based on the hybrid PSO-GSA optimisation In this section, we employ the hybrid PSO-GSA algorithm to optimise input weights and hidden layer biases of ELM, which mainly includes the following two parts: ELM and the PSO-GSA optimisation for ELM. 3.2.1 ELM Extreme Learning Machine (ELM) is proposed as a Single hidden Layer Feedforward neural Network (SLFN). Different from other neural networks, the connection weights between the input layer and the hidden layer and the biases of the hidden layer can be set randomly, and there is no need to adjust after setting. In addition, the output weights between the hidden layer and the output layer do not need to be updated iteratively, which can be determined once by solving the equations analytically. Compared with traditional neural networks, ELM has the following advantages: fast learning speed, good universal approximation performance and good generalisation performance. Figure 2 depicts the basic structure of ELM. Suppose that ELM with L hidden neurons is used to learn N arbitrary distinct instance distribution X , T = x i , t i i = 1 N , where x i = [ x i 1 , x i 2 , … , x i n ] T ∈ R n is the input vector, t i = [ t i 1 , t i 2 , … , t i m ] T ∈ R m is the target vector. In this paper, the output ti represents the ground truth for classification (1 denotes the defective instance, 0 denotes the non-defective instance). The output function of the ELM with L hidden neurons is as follows: ∑ i = 1 L β i ⋅ G ( w i ⋅ x j + b i ) = y j , j = 1 , 2 , … , N (9)where βi denotes the weight vector connecting the ith hidden node with output nodes, wi denotes the input weight matrix, bi represents the bias vector of the ith hidden node, y j = [ y j 1 , y j 2 , … , y j m ] T ∈ R m represents the jth output vector of ELM. G(.) denotes the activation function. In ELM, the parameters wi and bi can be randomly generated, and we use the hybrid PSO-GSA algorithm to optimise these two parameters. FIGURE 2Open in figure viewerPowerPoint The structure of extreme learning maching The above Equation (9) can be written compactly as: Y = H ⋅ β (10)where H denotes the output matrix of the hidden layer, Y represents the network output, and H, β, Y can be expressed as follows: H = G ( w 1 ⋅ x 1 + b 1 ) ⋯ G ( w L ⋅ x 1 + b L ) ⋮ ⋱ ⋮ G ( w 1 ⋅ x N + b 1 ) ⋯ G ( w L ⋅ x N + b L ) N × L (11) β = β 1 T β 2 T ⋮ β L T L × m a n d Y = y 1 T y 2 T ⋮ y N T = y 11 ⋯ y 1 m ⋮ ⋱ ⋮ y N 1 ⋯ y N m N × m (12) We use Sigmoid as the activation function of hidden neurons. The aim of ELM learning is to find the optimal output weight β by computing the least-squares solution, and the least-squares solution can be formulated as follows: m i n β ∈ R L × m H β − Y (13) The optimal solution of the Equation (13) is as follows: β ˆ = H ∔ (14)where H∔ denotes the Moore-Penrose generalised inverse of the hidden layer output matrix H. 3.2.2 The PSO-GSA optimisation for ELM In standard ELM, the input weights and hidden layer biases are set randomly, and these parameters may be not optimal for the defect prediction performance. As a consequence, to solve the above issue, we employ a hybrid PSO-GSA algorithm to search for the optimal input weights and hidden biases of ELM, and then output weights of ELM are determined analytically by the Moore-Penrose generalised inverse. The GSA is capable of quickly finding the global optimal solution of parameter combination by speeding up the convergence rate based on the initial parameters adjusted properly. Unfortunately, as the generation increases, the convergence rate that the GSA obtains the global optimal solution gradually slows down, and the GSA is easy to fall into the local optimal solution. In this paper, PSO is used in combination with GSA to effectively overcome the deficiency of GSA. The PSO algorithm has been widely applied and popularised due to its fast convergence property and simple evolutionary operators. The basic idea of the hybrid PSO-GSA is to integrate the exploitation capacity (fast convergence property) in PSO with the exploration capacity in GSA to synthesise both algorithms' strength. The hybrid PSO-GSA optimisation for ELM was mathematically modelled as follows: In the ELM network structure, input weights w and hidden layer biases b are represented by each agent which is considered as a candidate solution. We adopt the matrix encoding to encode every agent. From the Figure 2, we can observe w, and b as follows: w = w 11 ⋯ w n 1 ⋮ ⋱ ⋮ w 1 L ⋯ w n L , b = b 1 b 2 ⋮ b L (15)where w indicates the weights from the input layer to the hidden layer, b denotes the biases of the hidden layer. Given n agents in the hybrid PSO-GSA, each agent represents a potential candidate solution, that is, input weights and hidden layer biases of ELM. X i = ( x i 1 , x i 2 , … , x i d ) , i = 1 , 2 , … , n (16)where x i d is the position of the ith agent in the d-dimensional space. In this paper, the mean square error between the target output value and the predicted output value is used as the fitness function. The smaller the mean square error of the agent, the larger the fitness value and the better the agent. The equation of the fitness function is as shown in Equation (18): m s e = 1 N ∑ n = 1 N ( Y − T ) 2 (17) f i t n e s s = 1 1 + m s e (18)where mse is the mean square error of ELM between the target output value and the predicted output value, Y is the predicted output value of ELM, T is the target output value of ELM.