Software defect prediction via LSTM

IET SoftwareVolume 14, Issue 4 p. 443-450 Research ArticleOpen Access Software defect prediction via LSTM Jiehan Deng, Jiehan Deng School of Computer Science and Engineering, South China University of Technology, Guangzhou, People's Republic of ChinaSearch for more papers by this authorLu Lu, Corresponding Author Lu Lu lul@scut.edu.cn School of Computer Science and Engineering, South China University of Technology, Guangzhou, People's Republic of ChinaSearch for more papers by this authorShaojian Qiu, Shaojian Qiu School of Computer Science and Engineering, South China University of Technology, Guangzhou, People's Republic of ChinaSearch for more papers by this author Jiehan Deng, Jiehan Deng School of Computer Science and Engineering, South China University of Technology, Guangzhou, People's Republic of ChinaSearch for more papers by this authorLu Lu, Corresponding Author Lu Lu lul@scut.edu.cn School of Computer Science and Engineering, South China University of Technology, Guangzhou, People's Republic of ChinaSearch for more papers by this authorShaojian Qiu, Shaojian Qiu School of Computer Science and Engineering, South China University of Technology, Guangzhou, People's Republic of ChinaSearch for more papers by this author First published: 01 August 2020 https://doi.org/10.1049/iet-sen.2019.0149Citations: 14AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Software quality plays an important role in the software lifecycle. Traditional software defect prediction approaches mainly focused on using hand-crafted features to detect defects. However, like human languages, programming languages contain rich semantic and structural information, and the cause of defective code is closely related to its context. Failing to catch this significant information, the performance of traditional approaches is far from satisfactory. In this study, the authors leveraged a long short-term memory (LSTM) network to automatically learn the semantic and contextual features from the source code. Specifically, they first extract the program's Abstract Syntax Trees (ASTs), which is made up of AST nodes, and then evaluate what and how much information they can preserve for several node types. They traverse the AST of each file and fed them into the LSTM network to automatically the semantic and contextual features of the program, which is then used to determine whether the file is defective. Experimental results on several opensource projects showed that the proposed LSTM method is superior to the state-of-the-art methods. 1 Introduction Modern software is becoming more and more powerful, thus its scale and complexity continue to increase, which brought a great threat to the quality and reliability of software. To ensure software quality, companies have to employ quality assurance team to find defects in the software, which is a labour-intensive and costly work. However, to find as many bugs as possible, software testing often requires a large amount of time to perform various test cases, thus, with finite budgets and tight schedule, it is impractical to run the test for the entire project. Therefore, software defect prediction (SDP) has been proposed not only to reduce the cost and time for software testing, but also help the assurance team to locate the defective code more easily. And software defect prediction has attracted many researchers in recent years [1-4]. SDP is a process of building a defect prediction model using the historical data and then predict where the new code is defective. Previous studies on SDP mainly focussed on using either machine learning or deep learning techniques to build an effective defect prediction model. The former approach used handcrafted features like Halstead features [5], McCabe features [6], CK features [7], and build classifiers using machine learning techniques such as logistic regression (LR) or support vector machine, while the latter used deep learning techniques to construct powerful neural networks and made use of the program's Abstract Syntax Trees (ASTs) to build prediction models. And many studies [8-11] have proven the success usage of AST. Handcrafted features mainly focus on the statistical characteristics of the program, assuming that the buggy code has distinguishable features when compared to the clean ones. However, the handcrafted features of clean and buggy code can resemble, which will make it hard for classifiers to distinguish the difference. For example, Fig. 1a shows a non-buggy code snippet with two while loops. We simply switch the operation on the variable ‘i’ of the two while loops and now the file, shown in Fig. 1b is buggy as it will encounter an infinite loop. Fig. 1Open in figure viewerPowerPoint Two highly identical code snippets (a) non-buggy code, (b) buggy code due caused by switching operators However, the length of code, the number of operands and operators or even its complexity remains unchanged, which is hard to tell whether it is defective through those statistical features. And our observation on real-world programs does show that there are a number of files whose handcrafted features are very identical before and after the bugs are fixed. So in this paper, we focus out point on the program's semantic and contextual features. Like human languages, programming languages have to follow certain syntax rules, which lead to rich semantic and contextual information that can be extracted from the program's ASTs. AST is the representation [12] of source code, and contains the information of how the code is organised and interacts with each other. Again, in Fig. 1, we can simply know that the cause of the infinite loop is not only the operation on variable ‘i’ is ‘–’, but also ‘i < 10’ is the condition of the while loop and the result of ‘–’ operation on ‘i’ will always meet the condition of the for a loop. A simple form of the two files’ AST is shown in Fig. 2. The difference between the two ASTs is obvious – the operation on variable ‘i’, which shows where and what causes the different behaviour of the two code snippets. Such slight difference that leads to huge variance results is what handcrafted features failed to capture while it can be easily reflected by AST. So leveraging deep learning methods to mine this hidden information inside AST can lead to more accurate prediction results. Fig. 2Open in figure viewerPowerPoint Difference of the two ASTs (a) AST of the non-buggy code, (b) AST of the buggy code Recurrent neural networks (RNNs) [13] have been wildly applied in natural language process (NLP) like language modelling [14], machine translation [15], speech recognition [16], and video captioning [17]. RNN performs sequential processing by modelling units in sequence, and it can ‘memorise’ the information of previous computations. Since defective code is closely related to its context, thus RNNs are more suitable for capturing semantic and contextual features than other networks, such as CNN, which is more capable of detecting local features. We also employed the word embedding technique, which is also used in NLP tasks, to map the AST tokens into numerical vectors and help RNN to better capture the semantic and contextual features. To make good use of the semantic and contextual information inside the program's ASTs, this paper proposed a deep learning base SDP framework named defect prediction via LSTM (DP-LSTM), which can capture semantic and contextual features and use them to improve the performance of defect prediction. Specifically, we first parse the source code files into ASTs and traverse the AST to form the sequence of AST nodes for each file. Then we evaluate how much information we should preserve for a single AST node and transform them into numerical vectors using word embedding. The sequence of the AST node is then fed into the RNN to generate semantic and contextual features which are input to a Softmax classifier to determine whether the file is defective. To assess the proposed DP-LSTM approach, we explored the following two research questions: RQ1: Can DP-LSTM yield better performance than traditional methods that only use handcraft features to make prediction? RQ2: Compared to the state-of-the-art deep learning-based approaches, how effective is DP-LSTM approach? This paper makes the following contributions: We propose a defect prediction framework based on RNN to mine semantic and contextual information from the program's ASTs. Experimental results on seven open-source projects from PROMISE dataset show that the performance of DP-LSTM is better compared to the state-of-the-art deep learning-based method. We traverse the program's ASTs to form the AST node sequence for each source code file and apply dictionary mapping and word embedding techniques to convert the sequence into the input of RNN to capture the semantic and contextual information. The rest of the paper is summarised as follows. Section 2 introduces the related work on SDP. We elaborate on our DP-LSTM approach in Section 3. Experimental setup and result analysis are presented in Section 4. Finally, we conclude our work in Section 5. 2 Related work SDP has become one of the most popular research fields in software engineering [18]. Aiming to predict defect-prone software modules, it can help software quality assurance team to better allocate limited testing resources. Traditional approaches that only use handcraft features may fail to capture the information that is hidden inside the program's AST, so many studies now focus on using deep learning techniques to mine valuable information from AST and use them to more accurate prediction models. 2.1 Deep learning in software defect prediction Recent years, some researchers [8, 19] have suggested that only consider handcrafted features it is not enough to perform SDP accurately. To expand the features available in SDP and improve prediction performance, they tried to use deep-learning methods (e.g. deep belief network, DBN [20] and convolutional neural network, CNN [19]) to mine the rich semantic and structural features hidden inside the source code. The main idea of these methods is to extract semantic and structural features from the token vectors generated by programs’ ASTs and feed them to the machine-learning classifier to obtain the SDP model. Their experiments show that DBN and CNN methods are superior to the traditional SDP methods that use only handcrafted features in the SDP tasks. 2.2 LSTM network Long short-term memory (LSTM) network is a variation between RNN, which is specialised for processing sequences input like text and videos. Unlike other neural networks, RNNs can use their internal state (memory) to learn the relationship between the pieces of sequence, which makes it possible for RNNs to capture contextual features of the sequence. And RNNs have been proved successful in many NLP tasks like language modelling, machine translation, speech recognition, and image captioning. However, standard RNNs may encounter vanishing gradient problem [21] when the input sequence is too long, which make it impossible for RNNs to learn the connection from the information that has a longer gap. Thus, LSTM networks [22] are introduced to overcome the long-term dependency problem. In this paper, we leverage bidirectional LSTM for semantic and contextual feature generation from the program's ASTs. Fig. 3 shows the difference between RNN and LSTM units. Compared to standard RNN, a cell state and several cell state control gates are added to the neurons of LSTM, making LSTMs have long-term memory across the entire sequence. For standard RNNs, let denote the output of moment t, the loss function at the moment t is: and the total loss for the entire sequence is . Let denotes the input weights. According to the chain rule, the divertive of with respect to the loss function of any moment t is: (1)where denotes the state of the RNN unit at the moment j. If we use for the activation function of the neurons, we have: (2)Since the derivative value of lies between 0 and 1, and only when equals to 0, the divertive value is 1. However, during the training process, the divertive value of is usually smaller than 1 in and when t becomes larger, the product of tend to approach 0, which means will no longer be effective when t becomes larger. Fig. 3Open in figure viewerPowerPoint Difference between the unit of RNN and LSTM In LSTM, however, whether the information from previous cell state should be preserved is now controlled by the forget gate . By adding forget gate, the state of the neuron at the moment t now becomes: (3)For , nearly most of its value is either 0 or 1, so it is possible to control whether the information from previous cell state should be preserved through the forget gate. The effectiveness of LSTMs largely depends on its parameters such as hidden dimension and batch size. Thus, tuning is a key to train a successful LSTM. We will discuss how to tune these parameters in Section 6.2. 3 Methodology In this section, we elaborate our proposed DP-LSTM framework which can automatically generate semantic and contextual features from AST and utilise them for defect prediction. 3.1 Overall architecture Fig. 4 demonstrates the overall structure of our proposed SDP framework which automatically learns the semantic and contextual features from the source code. Specifically, the framework contains four steps: (i) source code parsing and sequence converting; (ii) AST node mapping and embedding; (iii) bidirectional LSTM building; (iv) defect prediction. Fig. 4Open in figure viewerPowerPoint Difference of the unit of RNN and LSTM 3.2 Source code parsing and sequence converting The input of out framework is labelled Java files from the previous version and unlabelled Java files from the newer version. Previous studies [23-25] have proven that program's AST contains rich information which can be mined and used for defect prediction, so we adopt AST as a high level representation of the program's source code. In this step, we use Javalang [26] to parse the Java files into ASTs and each line in Java file can be represented as one or several nodes in the AST. There are numerous types of tree nodes inside the AST, but not all of them relate to the defect of the code, so we evaluated the node types and Table 1 shows the categories and types of nodes we selected. In NLP tasks like text classification [27], words are treated as basic units in sentences, and a sentence is a sequence of multiple words. However, in a programming language, it is impractical to treat words as basic units [28] because the functional meanings of variables, methods and classes are not determined by their literal meanings but how they are used (e.g. we can use the variable ‘max’ to store the minimum value). Thus, we treat the AST node as the basic unit and we traverse the AST to get the sequence of AST nodes. Table 1. AST nodes we selected Node category Node type nodes of method invocations and instance creations MethodInvocation, SuperMethodInvocation, ClassCreator declaration-related nodes PackageDeclaration, InterfaceDeclaration, ClassDeclaration, ConstructorDeclaration, MethodDeclaration, VariableDeclarator, FormalParameter control-flow-related nodes IfStatement, ForStatement, WhileStatement, DoStatement, AssertStatement, BreakStatement, ContinueStatement, ReturnStatement, ThrowStatement, TryStatement, SynchronizedStatement, SwitchStatement, BlockStatement, CatchClauseParameter, TryResource, CatchClause, SwitchStatementCase, ForControl, EnhancedForControl other nodes BasicType, MemberReference, ReferenceType, SuperMemberReference, StatementExpression, 3.3 AST node mapping and embedding Since the AST node sequence cannot be directly fed into the neural network, we need to convert the AST node sequence into numerical vectors. In this step, we first build a unified dictionary mapping between nodes and integer values for the source and target project. Simply mapping the node type into an integer is applicable but, to some extent, may fail to capture important information from the AST node. Directly mapping the specific name of the node can capture every detail from the AST. However, the naming of variables, methods and classes is project specific and is closely related to the programmer's habit (e.g. the method that returns the maximum value can be named as ‘max’, ‘maxVal’ or ‘maxValue’). Such detailed information is redundant for defect prediction and may obfuscate the classifier. However, node information such as modifiers and variable types is vital to defect detection, because to some extent, they can lead to the defect of the code. For example, Fig. 5 shows two highly identical code snippets. Except the names of variables and methods, the only critical difference between them is the modifiers of the filed ‘sdf1’ (private) and ‘sdf2’ (private static). However, the second code snippet is highly defect-prone because the class ‘SimpleDateFormat’ is thread unsafe and the instances of class ‘Sample2’ share the same filed ‘sdf2’. If ‘method2’ of these instances are invoked in different threads concurrently, it is very likely to cause an exception. Such kind of defects is often hard to discover, especially in modern software modules where multi-thread programming is wildly used. Figs. 6 and 7 show the ASTs with plain node type and ASTs with the specific node name of the two code snippets (nodes that are different from the two code snippets are coloured as yellow, otherwise, are coloured as blue). Obviously, neither of the ASTs can accurately indicate what causes the defect in the second code snippet. Fig. 5Open in figure viewerPowerPoint Another highly identical code snippet Fig. 6Open in figure viewerPowerPoint AST with only node type Fig. 7Open in figure viewerPowerPoint AST with specific node name To achieve the balance between two kinds of AST, we go through the node type we selected in the previous step, and Table 2 shows the information we preserved for certain node types (other node types will be mapped into an integer using their type name). Fig. 8 shows the AST with information that we preserved, and the cause of the defect can be found. After the dictionary is built, we then use the word embedding technique to represent each token as a high-dimensional vector to let the network learn the relation between nodes. Table 2. Information we preserved for each node type Node type Information preserved MethodDeclaration modifiers, parameter types FieldDeclaration modifiers MethodInvocation method name, parameters’ name MemberReference member name ClassDeclaration modifiers VariableDeclaration variable type SynchronisedStatement lock object name IfStatement condition WhileStatement loop condition ForStatement loop condition Fig. 8Open in figure viewerPowerPoint AST with selected info 3.4 Bidirectional LSTM building In this step, we employed RNN to better utilise the semantic and contextual information of the AST. Standard RNN takes sequences as input, and each step of the sequence denotes a certain moment. For a certain moment t, the output not only depends on the current input but is also influenced by the output from the previous moment . The output of moment t can be formulated as the following equations: (4)where U, V, and W denote the weights of the RNN, b and c denote the bias. f and g are the activation functions of the neurons. However, as the length of the sequence continues to grow, standard RNN may encounter vanishing gradient problem due to the backpropagation mechanism, in other words, standard RNN can hardly learn the relationship from the information that requires long-term dependencies. To handle the vanishing gradient problem, we employed LSTM, which is a variation of RNN. Compared to standard RNN, LSTM can keep long-term memory by adding cell state and several gates to the RNN neurons. The cell state carries the information from the previous moments and will flow through the entire LSTM chain, which is the key that LSTM can have long-term memory. The gates of LSTM consists of a forget gate, an input gate and an output gate, which control how the information is kept in the cell states. The forget gate decides what information should be filtered from the previous moment, the output of forget gate can be formulated as the following equation: (5)where denotes the activation function, and denote the weights and bias of the forget gate, respectively. The input gate determines what information should be kept from the current moment, and its output can be formulated as the following equation: (6)where denotes the activation function, and denote the weights and bias of the input gate, respectively. With the information from forget gate and input gate, the cell state is updated through the following formula: (7) is a candidate value that is going to be added into the cell state and is the current updated cell state. Finally, the output gate decides what information should be outputted according to the previous output and current cell state (8)As stated above, the defective code is closely related to its previous and subsequent code segments, so we build a bidirectional LSTM to better handle the context dependencies. 3.5 Defect prediction Relying on the bidirectional LSTM, we can extract the semantic and contextual features from the AST node sequence and use it for defect prediction. In this step, we choose LR as the final classifier. We train the classifier using the previous version of the program, and then perform defect prediction on the newer version. 4 Experimental setup In this section, we describe the detailed settings for our experiments. 4.1 Research questions RQ1: Can DP-LSTM yield better performance than traditional methods that only use handcraft features to make predictions RQ2: Compared to the state-of-the-art deep learning-based approaches, how effective is DP-LSTM approach? 4.2 Experimental datasets To verify the validity of DP-LSTM approach, we chose seven open-source projects as our evaluation datasets. All these projects come from PROMISE repository which is a public dataset, and is used wildly in SDP studies. In our experiments, we extracted semantic and contextual features from Java source code using LSTM and adopted the labels from the PROMISE repository. Table 3 shows the detailed information of selected projects, including project name, project version, number of instances, and average defect rate (the percentage of defective instances). To guarantee the generality of the evaluation results, experimental datasets consist of projects with different sizes and defect rates (in ten projects, the maximum number of files is 965, and the minimum number of files is 32. In addition, the minimum defect rate is 6.3% and the maximum defect rate is 98.8%). Table 3. Detailed information for the projects we selected Project Description Version (train, predict) Avg. Files Avg. Defect Rate, % Xalan A library for transforming XML files 2.5, 2.6 844 47.3 Poi Libraries for reading and writing files in Microsoft Office formats 2.5.1, 3.0 413 64.2 Camel A framework for message-oriented middleware 1.4, 1.6 918 18.1 Jedit Programmer's text editor 4.0, 4.1 309 25.0 Lucene Ultra-fast Search Library and Server 2.0, 2.2 221 53.7 Log4j A Java logging tool 1.1, 1.2 149 76.5 Xercers An XML parser 1.2, 1.3 447 15.7 However, most of the datasets suffer from class imbalance, which is a common phenomenon is SDP [4, 29]. As shown in Table 2, the defect rate of most projects are neither too high nor too low and if the classifier is trained on a highly imbalanced dataset, the predictive model tends to support the majority class. And the ability to detect the minority class is weak. Many studies have proposed various approaches [4, 30, 31] to deal with class imbalance, but since it is not the key point of this paper, we adopt the standard random oversampling method – SMOTE [32-34] to handle the class imbalance problem. 4.3 Evaluation metrics There are many evaluation metrics [35] for the prediction model. Like other binary classification tasks, the result has four types: classifying an actual defective file as defective (true positive, TP); classifying an actual clean file as defective (false positive, FP); classifying an actual defective file as clean (false negative, FN); and classifying an actual clean file as clean (true negative, TN). We can construct a confusion matrix from these results and Table 4 demonstrates the confusion matrix. Table 4. Confusion matrix Actual positive Actual negative predicted positive TP FP predicted negative FN TN According to the confusion matrix, we can get the definition of Precision and Recall. Precision (also known as positive predictive value) is the fraction of actual positive instances among all instances that are predicted as positive: (9)Recall (also known as sensitivity) is a measurement of the integrity and defines the probability of the number of TP instances compared with the number of all positive instances (10)In practice, however, there is an inverse relationship between precision and recall, where it is possible to increase one at the cost of reducing the other. For example, if we demand a higher recall, we can predict all files as defective. Though recall will reach 100%, the precision will be very low. So F-measure is introduced to make a balance between precision and recall and it is the harmonic mean of the precision and recall: (11)For this reason, we adopt F-measure [36] as the evaluation metric of our model. F-measure is the most commonly used metric in many SDP studies. 4.4 Model comparison To further verify the validity of our approach, we chose several deep learning-based methods for comparison including: LR: An simple logistic regression classifier based on handcrafted features. DBN: Using deep-belief-network as features extractor to extract semantic features hidden on source code. Then training LR using those features. Following Wang et al.'s [8] parameter settings, we set 10 hidden layers and 100 nodes each layer. DBN + : Combining the sematic features extracted by DBN with the handcraft features. CNN: Giving a separate number for each AST node, then feeding it into the CNN. Referring to [37], the batch size is set as 32, the training epoch is 15, the filter length is 5 and the number of the hidden units is set as 100. DP-CNN: Introducing handcrafted features base on CNN. For fair comparations, we use LR as the classifier for all models. The implementation of LR is from sklearn.linear_model and we adopted the default parameters. Moreover, we follow the same approach to parse the source code into AST and convert to numerical vectors. To avoid the randomness caused by oversampling, we repeated the experiment for at least ten times and recorded the average performance. 5 Results Table 5 lists the F-measure results of all models on the selected projects and the highest values are marked as bold. Take project Xalan as an example, the F-measure value for LR, DBN, CNN are 0.344, 0.513 and 0.460, while DP-DBN and DP-CNN, DP-LSTM can achieve 0.634, 0.541 and 0.655, respectively. From Table 5, we can find that DP-CNN and DP-DBN achieve better results since they combined deep learning generated features and handcrafted features, while DP-LSTM can achieve an even better result without combining the handcrafted features. The average result in the last row indicates that DP-LSTM outperforms other deep learning-based methods. Table 5. F-measure comparison of different models Project LR DBN CNN DBN + DP-CNN DP-LSTM Xalan 0.344 0.513 0.460 0.634 0.541 0.655 Poi 0.472 0.561 0.710 0.639 0.729 0.735 Cammel 0.126 0.292 0.341 0.325 0.364 0.376 Jedit 0.249 0.397 0.436 0.472 0.459 0.445 Lucene 0.475 0.625 0.587 0.629 0.592 0.643 Log4j 0.308 0.472 0.434 0.516 0.481 0.527 Xerces 0.136 0.217 0.254 0.245 0.252 0.268 W/T/L 7/0/0 7/0/0 7/0/0 7/0/0 6/0/1 — average 0.301 0.411 0.460 0.480 0.488 0.521 5.1 RQ1: Can DP-LSTM yield better performance than traditional methods that only use handcraft features to make predictions DP-LSTM, which considers semantic and contextual features, could perform better than the traditional methods that only use handcrafted features as LR can only achieve an average performance of 0.301 which is lower than other deep learning-based approaches. 5.2 RQ2: compared to the state-of-the-art deep learning-based approaches, how effective is DP-LSTM approach? As Table 5 shows, the results of the approach wins six times compared to other models. Despite the performance of DP-LSTM is not always best, the average F-measure of DP-LSTM is 0.521 which outperforms DBN, CNN, DP-DBN, DP-CNN by 26.7, 13.2, 8.5, and 6.7%, respectively. 6 Discussion 6.1 Why does DP-LSTM work? Experimental results from Section 5 show that our proposed DP-LSTM method which can make connections between s