Empirical studies on the impact of filter‐based ranking feature selection on security vulnerability prediction

IET SoftwareVolume 15, Issue 1 p. 75-89 ORIGINAL RESEARCH PAPEROpen Access Empirical studies on the impact of filter-based ranking feature selection on security vulnerability prediction Xiang Chen, Corresponding Author xchencs@ntu.edu.cn School of Information Science and Technology, Nantong University, Nantong, China Guangxi Key Laboratory of Trusted Software, Guilin University of Electronic Technology, Guilin, China Correspondence Xiang Chen, School of Information Science and Technology, Nantong University, Nantong, China. Email: xchencs@ntu.edu.cnSearch for more papers by this authorZhidan Yuan, School of Information Science and Technology, Nantong University, Nantong, ChinaSearch for more papers by this authorZhanqi Cui, Computer School, Beijing Information Science and Technology University, Beijing, ChinaSearch for more papers by this authorDun Zhang, School of Information Science and Technology, Nantong University, Nantong, ChinaSearch for more papers by this authorXiaolin Ju, School of Information Science and Technology, Nantong University, Nantong, ChinaSearch for more papers by this author Xiang Chen, Corresponding Author xchencs@ntu.edu.cn School of Information Science and Technology, Nantong University, Nantong, China Guangxi Key Laboratory of Trusted Software, Guilin University of Electronic Technology, Guilin, China Correspondence Xiang Chen, School of Information Science and Technology, Nantong University, Nantong, China. Email: xchencs@ntu.edu.cnSearch for more papers by this authorZhidan Yuan, School of Information Science and Technology, Nantong University, Nantong, ChinaSearch for more papers by this authorZhanqi Cui, Computer School, Beijing Information Science and Technology University, Beijing, ChinaSearch for more papers by this authorDun Zhang, School of Information Science and Technology, Nantong University, Nantong, ChinaSearch for more papers by this authorXiaolin Ju, School of Information Science and Technology, Nantong University, Nantong, ChinaSearch for more papers by this author First published: 09 December 2020 https://doi.org/10.1049/sfw2.12006Citations: 3AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Security vulnerability prediction (SVP) can construct models to identify potentially vulnerable program modules via machine learning. Two kinds of features from different points of view are used to measure the extracted modules in previous studies. One kind considers traditional software metrics as features, and the other kind uses text mining to extract term vectors as features. Therefore, gathered SVP data sets often have numerous features and result in the curse of dimensionality. In this article, we mainly investigate the impact of filter-based ranking feature selection (FRFS) methods on SVP, since other types of feature selection methods have too much computational cost. In empirical studies, we first consider three real-world large-scale web applications. Then we consider seven methods from three FRFS categories for FRFS and use a random forest classifier to construct SVP models. Final results show that given the similar code inspection cost, using FRFS can improve the performance of SVP when compared with state-of-the-art baselines. Moreover, we use McNemar's test to perform diversity analysis on identified vulnerable modules by using different FRFS methods, and we are surprised to find that almost all the FRFS methods can identify similar vulnerable modules via diversity analysis. 1 INTRODUCTION A large number of security vulnerabilities are reported every year, and these security vulnerabilities have imposed significant damages to individuals and companies. However, finding security vulnerabilities is a challenging task and needs an understanding of both the software under test and the attackers’ mindset [1]. Recently, methods using machine learning are increasingly popular in security vulnerability analysis and discovery, and this kind of method has received more interest [2]. Motivated by the previous studies on software defect prediction (SDP) [3, 4], security vulnerability prediction (SVP) aims to construct models via machine learning to identify potentially vulnerable program modules similarly. For modules extracted from projects, features used to measure these modules are vital to constructing high-quality SVP models. In a recent study, Walden et al. [5] considered two kinds of features. One kind considered traditional software metrics (SM) (such as lines of code [LOC], cyclomatic complexity) designed from the SDP research domain as features, and the other kind used text mining to extract term vectors as features. However, considering these two kinds of features can eventually generate a large number of features. Moreover, not all the features are beneficial to the construction of SVP models, and this problem is called the curse of dimensionality. In most cases, data sets with high dimensionality can result in the high computational cost of model construction and performance degradation for constructed models. In our investigated SVP data sets, we find the curse of dimensionality problem is severe after using principal component analysis. Therefore, using feature selection, which is an effective method for dimensionality reduction, to improve the performance of SVP has a critical research significance. To analyse the impact of feature selection on SVP performance, we consider three real-world large-scale web applications as our empirical subjects. These subjects have 3466 modules (i.e. files) and 223 vulnerabilities (such as code injection vulnerabilities and cross-site request forgery vulnerabilities). By considering traditional SM and term vectors extracted by text mining as features, the number of features for these gathered SVP data sets ranges between 3898 and 18,318. Existing feature selection methods can be classified into three categories: filter-based ranking methods, filter-based subset methods, and wrapper-based subset methods [6]. Due to the excessive number of features in gathered SVP data sets, we only consider filter-based ranking feature selection (FRFS) methods in our study. Compared to the other categories of feature selection methods, FRFS methods have much less computational cost because the latter will spend a lot of time on searching for the optimal feature subsets in a large-scale search space [7]. In our empirical studies, we consider seven different feature ranking methods (i.e. rankers) in total for FRFS and use a random forest (RF), which shows competitiveness performance in a previous study [5], as the classifier to construct SVP models. We mainly investigate the following two research questions. 1.1 RQ1: Can using FRFS methods improve the performance of SVP? In this RQ, we first find that using FRFS methods can improve the SVP performance no matter what ranker is used on our chosen web applications compared to three state-of-the-art baseline methods. The first two baseline methods adopted by Walden et al. [5] use only SM or only term vectors extracted by text mining to construct SVP models. The last baseline method uses both SM and term vectors extracted by text mining as features to construct SVP models. We then find that previous suggestions on the optimal feature selection ratio for SDP [6, 8-10] are not valid for SVP. Therefore, the optimal feature selection ratio should be carefully chosen according to the characteristics of gathered SVP data sets. Later, after analysing the selected features by our FRFS methods, we suggest that traditional SM and features extracted by text mining should be both considered. 1.2 RQ2: Can different FRFS methods identify the same vulnerable modules? In this RQ, we want to investigate whether there exists a difference between two SVP models using different FRFS methods in terms of the specific vulnerable modules each FRFS method identifies and does not identify. In particular, we use McNemar's test to perform diversity analysis on identified vulnerable modules. The final results show that the prediction diversity phenomenon on vulnerable modules for different FRFS methods does not exist in most cases. Moreover, we analyse the number of vulnerable modules which cannot be correctly identified by any FRFS methods. Final results show performance bottleneck exists in state-of-the-art SVP methods since any of our considered SVP methods cannot identify some of the vulnerable modules. Therefore, we should design more powerful SVP methods (such as using deep learning to learn semantic features from source codes in the modules automatically) in the future. The main contributions of this paper can be summarized as follows: To the best of our knowledge, we are the first to perform a depth analysis of the impact of FRFS on SVP. We consider seven different feature ranking methods in total from three different FRFS categories and then use a RF to construct SVP models Empirical studies are designed based on three real-world web applications, and final results show the effectiveness of applying FRFS to SVP compared to three state-of-the-art baseline methods. We also use McNemar's test to perform diversity analysis on identified vulnerable modules when comparing different FRFS methods 1.3 Paper organization The rest of this paper is organized as follows: Section 2 analyses the background of SVP and the motivation of our study. Section 3 shows our case study setup. Section 4 gives case study results. Section 5 discusses potential threats to validity analysis. Section 6 summarizes related work for our study. Section 7 concludes this paper and discusses some potential future work. 2 BACKGROUND AND MOTIVATION In this section, we first introduce the background of SVP and analyse the difference between SVP and SDP. Then we emphasize the motivation of our study and the design motivation of research questions. 2.1 Background of SVP SVP resorts to machine learning and aims to identify vulnerable modules in advance. Therefore, the allocation of limited resources for security audits and code inspection can be optimized. Then more security vulnerabilities may be detected and mitigated as soon as possible. The brief process of SVP can be summarized as follows [5]. SVP first extracts and labels program modules from software historical repositories, such as version control systems, bug tracking systems, National Vulnerability Database (NVD)s. The granularity of modules can be set to source code file, object-oriented class, binary component as needed. Then it uses features to measure these modules. Later, it labels a program module vulnerable if this module contains at least one vulnerability by analysing commit messages and bug reports in the bug tracking system or NVDs. Finally, it uses a specific classification method (i.e. RF) to train a model based on the gathered SVP data set. For a new program module, we can use the same metrics to measure this module and then use the trained model to predict whether this module is vulnerable or non-vulnerable. Based on the SVP process analysis, it is not hard to find that this research topic is mainly motivated by SDP [4, 11-17]. Vulnerabilities and defects have a certain similarity [2]. They are both caused by human mistakes, and these mistakes may be related to the code complexity or the developer experience. Therefore, most of SVP studies used metrics designed in SDP to construct SVP models and showed the feasibility of this solution. However, different from defects, vulnerabilities are instances of errors in the specification, development, or configuration of software such that their executions can implicitly or explicitly violate security policies. Therefore, vulnerable modules should have different characteristics when compared to defects, and we should design specific metrics to focus on these characteristics. Moreover, the number of vulnerable modules is far less than the number of defective modules in most projects [18]. Therefore, the problem of class imbalance in SVP is more challenging than SDP. 2.2 Motivation 2.2.1 Data set redundancy analysis In our study, we use principal component analysis (PCA) to analyse data redundancy in gathered SVP data sets (The details of these data sets can be found in Section 3.2) [6]. Figure 1 shows the PCA analysis result for PHPMyAdmin data set. In this figure, we sort principal components (PCs) in the descending order according to their explained variance. For example, c1∼c20 denotes the top 20 PCs, and they can explain 61.543% of the variance. To explain 95% of the variance, we only need the top 2.29% of PCs. The line in Figure 1 shows the cumulative variance explained by the PCs. It is not hard to find that most of the variance is contained in a small number of PCs. The same findings can be found in the remaining two gathered SVP data sets (i.e. Drupal and Moodle). Therefore, all the SVP data sets used in our empirical studies have certain data redundancy. FIGURE 1Open in figure viewerPowerPoint Data set redundancy analysis for PHPMyAdmin by using principal component analysis We consider feature selection methods to reduce the data redundancy in gathered SVP data sets, which can be used to perform dimensionality reduction. Existing feature selection methods can be classified into three categorizes (i.e. filter-based ranking methods, filter-based subset methods and wrapper-based subset methods) [6, 19, 20]. In particular, filter-based ranking methods select features by estimating their capacity for contributing to the fitness of the constructed models. Filter-based subset methods select features that collectively have a good prediction performance. Wrapper-based subset methods use a predetermined classifier and a performance measure to evaluate the importance of a feature subset for constructed models. As we consider both traditional metrics and term vectors extracted by text mining as features in our study, the final gathered SVP data sets eventually have 3898∼18,318 features. Therefore, we do not consider filter based subset methods and wrapper-based subset methods, since the computational cost of these methods is prohibitive [7]. Therefore, we want to analyse the first research question as follows. Previous studies on SDP [6, 19, 21] show that using feature selection can improve the performance of SDP. However, whether using FRFS methods can improve the performance of SVP has not been thoroughly analysed. Then, for SDP, previous studies [6, 8, 9] using FRFS methods for dimensionality reduction suggested selecting top log2N features from the list of ranked features, where N is the total number of features. For SVP, a recent study [10] suggested using top-10 or top-100 features. However, whether these recommendations on optimal feature selection ratio are valid in our empirical studies is still unknown and needs further investigation. Finally, we want to analyse the proportion of different types of features in the selected features since we use both SM and text features (TF) to measure the extracted modules. In the first research question, we mainly compare and rank the performance of different FRFS methods. In the second research question, we want to conduct an in-depth analysis through the analysis of vulnerable modules identified by SVP models using different FRFS methods. We want to investigate whether there exists a difference between two SVP models using different FRFS methods in terms of the specific vulnerable modules each FRFS method identifies and does not identify. Moreover, we also want to find the vulnerable modules which cannot be predicted correctly by all the FRFS methods. 3 CASE STUDY SETUP In this section, we first illustrate our case study approach. Then we show the studied data sets. Later we introduce the details in measuring extracted modules, performing data preprocessing, performing cross-validation. Finally, we analyse the measures in performance evaluation and statistical analysis methods for answering RQ1 and the diversity analysis method on identifying vulnerable modules for answering RQ2. 3.1 Approach The overview of our case study approach can be found in Figure 2. FIGURE 2Open in figure viewerPowerPoint The overview of our case study approach In this approach, we first use cross-validation to generate the training set and the test set (in Section 3.5), respectively. Then we consider two types of metrics to measure the extracted program modules (in Section 3.3). Later, we perform data preprocessing on the training set, including the feature selection method and the class imbalanced method. Finally, we can construct the SVP models by a RF classifier. The details of data preprocessing and model construction can be found in Section 3.4. We use this approach to answer two research questions in Section 2.2 (i.e. performance evaluation and diversity analysis on identifying vulnerable modules). 3.2 Experimental subjects In our empirical study, we choose the experimental subjects based on the following criteria: (1) The applications should be open-source projects, which we can access their source code and measure the extracted program modules. (2) The applications should be web applications written in PHP, since PHP is the most popular programming language in Web Development. (3) The applications should contain at least 20 detected vulnerabilities. Based on the above criteria, we consider three real-world open-source web applications written in PHP programming language. The selected web applications are Drupal, Moodle and PHPMyAdmin. In particular, Drupal is a widely used content management system, Moodle is an open-source learning management system, and PHPMyAdmin is a web-based management tool for MySQL database. Notice, in our study, we only consider these vulnerabilities related to a specific version of these applications (i.e. Drupal 6.0, PHPMyAdmin 3.3, and Moodel 2.0). In these web applications, the granularity of the extracted modules is set to file. Finally, 3466 modules are extracted, and 223 vulnerabilities are identified in total. Security vulnerabilities are mined for each application from vulnerability databases. In particular, for the application Moodle, the data source was the NVD1, while for the applications Drupal and PHPMyAdmin, the security announcements maintained by those projects are used. The detected vulnerabilities for these web applications have a variety of types, such as code injection vulnerabilities, cross-site request forgery vulnerabilities, cross-site scripting vulnerabilities and path disclosure vulnerabilities. 3.3 Measuring extracted modules Extracted modules can be measured in two different ways. One way uses traditional SM inspired by traditional studies in SDP (i.e. SM). The other way uses text mining to extract term vectors as features (i.e. TFs). 3.3.1 Software metrics Here 12 different SM are considered: LOC, LOC for non-HTML, Number of functions (NM), cyclomatic complexity, maximum nesting complexity, Halsteads volume, total external calls, fan-in, fan-out, internal functions/methods called, external functions/methods called and external calls to functions/methods. We introduce the details and abbreviation (i.e. corresponding metric name in Weka arff file) of each metric as follows: Lines of code This metric measures the number of lines in a source file excluding lines, such as blank lines and comments. LOC for non-HTML This metric is the same as the metric LOC. It measures the number of lines in a source file, except HTML content embedded in from which 3,466 modules (files) (i.e. content outside of php start/end tags). Number of functions This metric measures the number of function/method definitions in a source file. Cyclomatic complexity This metric measures the size of a control flow graph after linear chains of nodes are collapsed into one. Maximum nesting complexity This metric measures the maximum depth to which loops and control structures in the file are nested. Halsteads volume This metric measures the volume of one source file that can be computed by (N1 + N2)log(n1 + n2). Here n1 and n2 denote the number of distinct operators and the number of distinct operands, respectively. N1 and N2 denote the total number of operators and the total number of operands, respectively. To measure this metric, operators are method names and PHP language operators, while operands are parameter and variable names. Total external calls (NIC) This metric measures the number of instances where a statement in the source file being measured invokes a function/method defined in another source file. Fan-in (NICU) This metric measures the number of source files (excluding the source file being measured), which contain statements that invoke a function/method defined in one source file being measured. Fan-out (NOEFCU) This metric measures the number of source files (excluding the source file being measured), which contain functions/methods invoked by statements in one source file being measured. Internal functions/methods called (NOIC) This metric measures the NM/methods defined in the source file being measured which are called at least once by a statement in one same source file. External functions/methods called (NOEFC) This metric measures the NM/methods defined in other source files which are called at least once by a statement in one source file being measured. External calls to functions/methods (NOECU) This metric measures the number of source files (excluding the source file being measured) calling a particular function/method defined in the source file being measured, summed across all functions and methods in the source file being measured. 3.3.2 Text features Term vectors as TFs are extracted by text mining. The tokens and their associated frequencies of each PHP program module (i.e. file) are extracted by PHP's built-in function token_get_all. These tokens represent language keywords, punctuation, and other code constructs. Notice, firstly, comments and whitespace are ignored. Secondly, string and numeric literals are converted into fixed tokens. For example, T_STRING is used to represent a string literal instead of the string contents. 3.3.3 Statistics It is not hard to find that these two ways of measuring extracted program modules do not consider programming language characteristics. Therefore, these metrics are programming language independent and general. The statistics of these data sets can be found in Table 1. These characteristics include the project name, the number of extracted modules, the number (percentage) of vulnerable modules, and the number of features. These web applications have been used in previous SVP studies [5, 10, 22, 23], and the representativeness of chosen data sets can be guaranteed. TABLE 1. The statistics of security vulnerability prediction data sets in our empirical studies Dataset # Files #(%) Vulnerable # Text # Software Files Features Metrics Drupal 202 62 (30.69%) 3886 12 PHPMyAdmin 322 27 (8.39%) 5232 12 Moodle 2942 24 (0.82%) 18,306 12 3.4 Performing data preprocessing and constructing models 3.4.1 FRFS methods There are three categories for FRFS methods: statistics-based methods, probability-based methods and instance-based methods [6, 19, 21]. In our study, we consider seven different FRFS methods (i.e. two rankers in statistic-based methods, three rankers in probability-based methods and two rankers in instance-based methods). The details of these different rankers, which include category, name, description and abbreviation, can be found in Table 2. TABLE 2. Rankers considered by filter-based ranking feature selection methods Category Name Description Abbreviation Statistics based ChiSquared [24] This ranker evaluates the importance of a feature by computing the ChiSquared statistic with respect to the class label Ranker1 F-score [25] This ranker evaluates the importance of a feature by computing the F-score. The importance is measured by the distinguishing between two classes with real values Ranker2 Probability based GainRatio [26] This ranker evaluates the worth of a feature by measuring the gain ratio with respect to the class. GainRatio penalizes multi-valued features to mitigate the bias of InfoGain Ranker3 InfoGain [27] This ranker evaluates the worth of a feature by measuring the information gain with respect to the class. InfoGain is an entropy-based technique Ranker4 GiniIndex [28] This ranker evaluates the worth of a feature by measuring the Gini index with respect to the class Ranker5 Instance based FisherScore [28] This ranker evaluates the worth of a feature such that the feature values of instances within the same class are similar while the feature values of instances from different classes are dissimilar Ranker6 ReliefF [29] This ranker evaluates the worth of a feature by repeatedly sampling an instance and considering the value of the given feature for the nearest instance of the same and different classes Ranker7 Since we could not know the optimal feature selection ratio in advance, we will investigate different feature selection ratio from 5% to 95% and set the step size to 5%. For a specific feature selection ratio, we will construct the model M by using the selected features and evaluate the constructed model on the test set in terms of R performance measure (illustrated in Section 3.6). The pseudocode of determining optimal feature selection ratio can be found in Algorithm 1. Algorithm 1.FRFS for SVP Input: train: training data test: test data N: the number of original features classifier: classifier ranker: ranker used for FRFS Output: (R, ratio): a pair of the best R and the corresponding feature selection ratio 1: best R ← 0, best Ratio ← 0% 2: Sorting features in the descending order according to rankerbased on the training data train 3: Applying the class imbalance method to train 4:for ratioin {5%, 10%, ⋯, 95%}do 5: tmpR←0 6: Selecting top [ratio × N] features 7: Constructing the model M by using classifier based on train 8: Evaluating the model M on the test data test and store the performance of R to Rratio 9:if Rratio > best Rthen 10:bestR← Rratio, best Ratio ← ratio 11:end if 12:end for 13: return (best R, best Ratio) 3.4.2 Class imbalanced method Compared to defects, vulnerabilities are far fewer in the project and detecting vulnerabilities is similar to searching for a needle in a haystack [18]. In line with the previous study, we use a classical class imbalance method (i.e. SpreadSubsample) provided by Weka package and use the same setting suggested by Walden et al. [5]. 3.4.3 Classifier We use RF as the classifier to construct SVP models, since RF classifier is widely used in previous SVP studies and shows competitiveness performance [5, 23]. RF is an ensemble learning method for classification. It operates by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes [30]. 3.5 Performing cross-validation To evaluate the performance of trained models, we use 10×3-fold cross-validation (CV). The threefold CV process can be summarized as follows: The instances are randomly divided into threefolds with approximately equal size. Each fold has the same ratio of vulnerable modules as the entire data set. A model is trained based on the instances in two folds (i.e. the training set), and then the model is tested based on the instances in the remaining fold (i.e. the test set). This process is repeated three times and each fold is used as the test set at least once. The threefold CV is repeated 10 times to overcome the effect of randomness. The reason for not using the 10-fold CV is that the number of security vulnerabilities in some applications is too small. For example, the Moodle application only contains 24 vulnerabilities. If considering the 10-fold CV used by [10, 23], each fold can only have 2∼3 vulnerabilities, which is not very helpful for building a high-quality SVP model. 3.6 Performance measures and statistical analysis methods 3.6.1 Performance measures According to the ground truths and the predicted class labels, we can compute the number of true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN), respectively based on the confusion matrix in Table 3. Then, we can compute the precision measure P and the recall measure R as follows: P = T P T P + F P (1) R = T P T P + F N (2) TABLE 3. Confusion matrix based on the actual type and the predicted type of program modules Actual type Predicted type Vulnerable Non-vulnerable Modules Modules Vulnerable modules T P F N Non-vulnerable modules F P T N The measure P denotes the proportion of modules that are correctly classified as vulnerable modules among those predicted as vulnerable. The measure R denotes the proportion of vulnerable modules, which are correctly predicted. Considering the significant impact of vulnerable modules, the measure R is more important than the measure P in SVP. Because once some modules with vulnerabilities are not detected, they will cause huge losses after the project is deployed. It is not hard to find that the measure R is considered from