Stochastic gradient boosting for predicting the maintenance effort of software‐intensive systems

IET SoftwareVolume 14, Issue 2 p. 82-87 ArticleOpen Access Stochastic gradient boosting for predicting the maintenance effort of software-intensive systems Sergio Cerón-Figueroa, Sergio Cerón-Figueroa Laboratorio de Cómputo Inteligente, Centro de Investigación en Computación, Instituto Politécnico Nacional, Av. Juan de Dios Bátiz, México, MexicoSearch for more papers by this authorCuauhtémoc López-Martín, Corresponding Author Cuauhtémoc López-Martín cuauhtemoc@cucea.udg.mx Department of Information Systems, Universidad de Guadalajara, Periférico Norte # 799, Zapopan, Jalisco, MexicoSearch for more papers by this authorCornelio Yáñez-Márquez, Cornelio Yáñez-Márquez Laboratorio de Cómputo Inteligente, Centro de Investigación en Computación, Instituto Politécnico Nacional, Av. Juan de Dios Bátiz, México, MexicoSearch for more papers by this author Sergio Cerón-Figueroa, Sergio Cerón-Figueroa Laboratorio de Cómputo Inteligente, Centro de Investigación en Computación, Instituto Politécnico Nacional, Av. Juan de Dios Bátiz, México, MexicoSearch for more papers by this authorCuauhtémoc López-Martín, Corresponding Author Cuauhtémoc López-Martín cuauhtemoc@cucea.udg.mx Department of Information Systems, Universidad de Guadalajara, Periférico Norte # 799, Zapopan, Jalisco, MexicoSearch for more papers by this authorCornelio Yáñez-Márquez, Cornelio Yáñez-Márquez Laboratorio de Cómputo Inteligente, Centro de Investigación en Computación, Instituto Politécnico Nacional, Av. Juan de Dios Bátiz, México, MexicoSearch for more papers by this author First published: 01 April 2020 https://doi.org/10.1049/iet-sen.2018.5332Citations: 5AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract The maintenance of software-intensive systems (SISs) must be undertaken to correct faults, improve the design, implement enhancements, adapt programmes such that different hardware, software, system features, and telecommunications facilities can be used, as well as to migrate legacy software. A lack of planning has been identified as one explanation for late and over budget software projects. An activity of planning is effort prediction. The goal of this study is to propose the application of a stochastic gradient boosting (SGB) model for predicting the SIS maintenance effort. We compare the SGB prediction accuracy with those obtained with statistical regression, neural network, support vector regression, decision trees, and association rules. We trained and tested the models with five SIS data sets selected from the International Software Benchmarking Standards Group Release 11. The SGB prediction accuracy was statistically better than the mentioned five models in the two larger data sets. We can conclude that a SGB can be applied to predict the maintenance effort of SISs coded in languages of the third generation and developed on either mainframes or multi-platform. The predicted effort corresponds to the aggregate of efforts obtained from the project team, project management, and project administration. 1 Introduction A software-intensive system (SIS) is any system where software contributes essential influences to the design, construction, deployment, and evolution of the system as a whole [1]. A SIS includes (i) in-house systems for accounting, production control, and business, (ii) applications sold commercially in the mass market, and (iii) large, complex, military, industrial, and commercial systems [2]. The development of a SIS involves the design, construction, and testing, whereas software maintenance (SM) has its goal to modify existing software while preserving its integrity to ensure that the software continues to satisfy user requirements [3]. SM is formally defined by the IEEE Std. 1219–1993 as 'the modification of a software product after delivery to correct faults, improve the performance or other attributes, or adapt the product to a modified environment' [4]. SM activities are needed to maintain thousands of projects that have been reported [5] or analysed [6]. SM can account for over 60% of all effort expended by an organisation, and the percentage continues to rise as every year software is produced [7]: in the 1970s, the costs were around 60%, while in the 1990s and 2000s, the cost increased to about 90% and more [8]. An analysis done in the year 2016 involving software projects developed in countries such as the USA, Europe, and the rest of the world, revealed that only 29% of projects were successful, 52% of them were challenged, and 19% failed [6]. One of the ten highest project success factors is the presence of proper planning, whereas one of the ten highest project failed factors is lack of planning [9]. Software project planning involves software prediction [3]. It is important to distinguish between SM and software maintainability [10]. The latter is defined as 'the ease with which a software system or component can be modified to correct faults, improve the performance or other attributes, or adapt to a changed environment' [11]. Since practitioners who interact with SISs need predicting their resources needed to perform development, maintenance, operation, and management [2], our study is related to SM effort prediction (SMEP) for SISs; i.e. it tries to answer the following question: 'how many efforts (person-hours) are needed to modify a SIS after its delivery?'. The effort is predicted once the software requirements have been specified for either developing or maintaining software [3]. One of the two key issues leading to the importance of SM is its cost [12], and accurate effort prediction is needed for getting a better estimation of costs, schedule, and the risks of a software project [13]. Based upon SM consumes a major share of the financial resources in a software life cycle, SMEP is an important aspect in the planning phase [3]; however, to date, we only identified 16 SMEP studies published since 1995 [12-27]. Our study proposes a model applied to SMEP. This model is termed stochastic gradient boosting (SGB). A SGB can learn complex (non-linear) functions, and non-linear relationships are common between dependent and independent variables in software projects [28]. It is common practice to use software size as the independent variable for predicting project effort [29-31], and the size of a software product is mainly measured in function points (FPs) or lines of source code [32]. There exist unadjusted FPs (UFPs) and adjusted FPs (AFPs). UFPs are calculated from two data functions and five transactional functions, whereas AFP corresponds to UFP adjusted taking into account 14 systems' characteristics [13]. The contribution of our study is to propose the application of a SGB model for SIS maintenance effort prediction. A study on software effort prediction models published in 2018 recommends using the same data sets and accuracy measure [33], thus, we investigate if the accuracy obtained by the SGB (when AFP is used as the independent variable) is better than those reported from the following prediction models trained and tested with the same SIS data sets used in our study: Simple linear regression (SLR) [13, 14]. Feedforward multilayer perceptron (MLP), radial basis function neural network (RBFNN), and general regression neural network (GRNN) [13]. Two types of support vector regression (ɛ-SVR and ʋ-SVR) with four types of kernels by SVR (linear, polynomial, radial basis function, and sigmoid) [14]. Decision trees (DT) [14]. Association rules (ARu) [14]. We selected a statistical regression (SR, i.e. SLR) to be compared with the rest of the models because a guideline for acceptable comparative studies related to software effort prediction establishes that any new proposed prediction model should at least outperform a statistical model [34]. We perform the comparison among prediction accuracies of the SGB, SLR, MLP, RBFNN, GRNN, ɛ-SVR, ʋ-SVR, DT and ARu models taking into account the following issues suggested when machine learning models and non-machine learning models (such as a SLR) are compared [13] Use of the absolute residuals (ARs) as a criterion for comparing the prediction accuracy between the models. Application of the leave-one-out cross-validation (LOOCV) as a validation method. Description of the manner of how the SGB was optimised. Use of mutually exclusive data sets for training and testing the models. Selection of a suitable statistical test to compare the prediction accuracy among models from data dependence and normality analysis. Statistical significance difference as the means for comparing the prediction accuracy. A systematic review published in 2016 regarding an international repository including data of thousands of software projects, concluded that several studies lack clarity on how the data were prepared and used, which makes difficult to compare results among studies as well as replicate them [35]. Another study published in 2015 stands out that research studies in SM are notoriously hard to reproduce due to lack of datasets, and implementation details such as parameter values or environmental settings [36]. Finally, a study of 2018 found that many studies hinder replications and decreased the comparability among them because of unavailable data [37]. Thus, in this study, we used those public data sets, as well as the accuracy criterion and validation method used in two previous studies [13, 14]. The data sets of SISs used in our study were selected from the International Software Benchmarking Standards Group (ISBSG) Release 11. A type of SIS is business application [2]. Most SISs contained in the ISBSG have a business focus involving telecommunications (25%), banking (12%), insurance (12%), finance (8%), manufacturing (8%), engineering (5%), accounting (4%), and sales and marketing (4%) [5]. The SISs of ISBSG were developed in 24 countries [5]. The hypothesis to be tested in our research is the following: H1: The maintenance effort prediction accuracy for SISs with a SGB is statistically better than the accuracies obtained with SLR, MLP, RBFNN, GRNN, ɛ-SVR, ʋ-SVR, DT and ARu when adjusted FP data is used as the independent variable. The rest of this study is organised as follows: Section 2 presents related work on SMEP. Section 3 describes a SGB and its components. Section 4 presents the criteria for selecting the data sets from the ISBSG data set. Section 5 describes the training and testing for SGB. Section 6 compares the prediction accuracy results obtained for the models, whereas Section 7 presents a discussion, including our conclusions, the limitations of our study, validation threats, and future work. 2 Related work In spite of SM corresponds to the longest phase of the software life cycle and, in most cases, the most expensive [8], SM has not received the same degree of attention that the other phases have [3]. Regarding SMEP, we only identified two studies in the last five years [13, 14], 13 studies between 2001 and 2013 [12, 15-26], and one of 1995 [27]. The models applied in these 16 studies were statistical regressions [12-18, 21-23, 25-27], neural networks [13, 15, 17-19, 27], SVR [14], rule-based systems [14, 15], DT [14], Bayesian network [20], analogy [24], and a pattern recognition approach termed optimised set reduction [27]. In one of the two studies that use the same datasets than our reports that there was no statistically significant difference among the accuracies of a SLR and three types of neural networks [13] when a data set was used, whereas the second one reports that the type of ɛ-SVR was statistically better than SLR, three types of neural networks, ARu, and DT, for one of the five data sets used [14]. 3 Stochastic gradient boosting (SGB) 3.1 Bagging and boosting Bagging is an ensemble method for classification and regression developed by Breiman [38]. Its aim is to improve the performance of a base algorithm by training multiple instances of it with different random bootstrap samples of the training data set. Boosting combines multiple versions of a weak base algorithm to assemble a stronger version that exhibits better performance [39]. Boosting-based algorithms are trained using the entire data set with the peculiarity that, after each iteration, the incorrectly classified patterns are weighted more heavily. Training is then repeated considering the new weights. Boosting algorithms are essentially different to bagging, due to the fact that the former are trained using the complete, original data set; the latter trains the base algorithm using random samples with replacement of the dataset. It means that boosting has a reduced bias compared to bagging, in which performance does not greatly improve as the number of training patterns grows [40, 41]. 3.2 Gradient boosting (GB) The original GB method proposed by Friedman in 1999 [42] aims to minimise a loss function chosen arbitrarily, so long as it is differentiable using the gradient descent technique. The basic idea behind this method is to add the best estimator h(x, a) to the base model F(x) in the opposite direction of the gradient in such a way that the loss function is reduced with each iteration. The original GB algorithm is described as follows [41]: For m = 1 to M do End For Where is the model created in each iteration, which improves the base algorithm F(x). 3.3 Base learner GB may use any base model to form the ensemble, including regression splines and generalised linear models. Friedman proposed the use of fixed-size classification and regression trees as a base model because of their simplicity and power [43, 44]. Assuming that h(x, a) is a terminal L-node of a tree, in each iteration the x space is partitioned into disjoint L-regions that predict a constant value for each region. Thus, the GB algorithm with regression trees is given as follows: For m = 1 to M do End For 3.4 Loss function Since GB is an iterative algorithm that seeks to minimise any differentiable loss function, there exist several measures that satisfy this requirement, the most common being least-squares, least-absolute-deviation (LAD) and Huber (M), among others [45]. As for regression problems, LAD is a considerably robust method because the approximation is calculated using the sum of the absolute values of the residuals between the predictions and the actual data, forming a trend line in the case of Cartesian coordinates. In the case of trees, the updating of the nodes is made based on the median as follows: 3.5 SGB algorithm SGB is an improvement over the original GB algorithm. SGB was proposed by Friedman as a modification that enables validation of the model by iteration [43]. It works by partitioning the training set into two random subsets, where one of them will serve as training data and fit the base learner. In contrast to bagging, the random sampling of the subsets is made without replacement, thus reducing variance in the results obtained by execution of the algorithm. The SGB proposed in our study refers to a tree-based SGB using the LAD loss function. Let the full training dataset and a random permutation of integers. Then, the SGB algorithm is the following: For m = 1 to M do End for Where L is the number of trees, N is the number of elements in the original training set, is the random subsample of size . In this stochastic model, the fraction of patterns used to train the base learner is defined as follows: The value of f can lie between 0.5 and 1, where a value of 1 means that the algorithm is the same as the deterministic version. Friedman recommends the use of a value of to mitigate the over-fitting effect [43]. In our tests, the best result was obtained by using . The stochastic version of the GB model, apart from improving performance, also speeds up the training of the base learner due to the use of fewer patterns. 4 Description of data sets of projects We selected five SIS data sets from the ISBSG Release 11 [46]. We performed a SR analysis on a data set consisting of a scatter plot (effort versus AFP), correlation and determination coefficients, as well as analysis of variance by using the equation as in [14]. The ISBSG contains 5052 SISs developed between 1989 and 2009. The size of a SIS is measured in AFP, whereas the effort in person-hours, which can be transformed to person-months (one person-month is equal to 152 h [31]). The criteria for selecting the SISs were based on the 'Guidelines for use of the ISBSG data' document, which suggests taking into account data quality, type of development, sizing method, development platform (DP), programming language generation, and resource level (RL) [46]. The ISBSG rates its data quality from 'A' to 'D' and the data sets of our study correspond to those having 'A' and 'B' because these types are suggested for statistical analysis [46]. The selected SISs were those ones whose type of development was an enhancement, which is defined as 'changes made to an existing application where new functionality has been added, or existing functionality has been changed or deleted' [47]. As for the sizing method, ISBSG includes SISs measured with several FPs' standards such as IFPUG, MARK II, NESMA, FiSMA, and COSMIC. The data sets used in our study consist of those SISs measured with the IFPUG V4+ since it is not possible to mix pre-IFPUG V4 with V4 and post-V4. Regarding DP, it refers to the operating system used such as personal computer, mid-range, mainframe, or multiplatform. In accordance with programming language generation, the ISBSG includes second (2GL), third (3GL), and fourth (4GL) generation, as well as application generator. Finally, the RL refers to people whose time is included in the reported effort data. ISBSG reports the following four levels (the number at each level indicates that all efforts at this and preceding levels are included in the effort fields): Development team effort: project team, project management, project administration. Development team support: database administration, data administration, quality assurance, data security, standards support, audit and control, technical support. Computer operations involvement: software support, hardware support, information centre support, computer operators, network administration. End users or clients: user liaisons, user training time, application users and/or clients. Table 1 describes the five data sets used in our study [14]. Table 1. Data sets of SISs () DP LT RL Size MF 3GL 1 267 MF 3GL 4 50 MR 3GL 1 39 MP 3GL 1 102 MP 4GL 1 65 Since data depicted in the scatter plots from Table 1 show skewness, heteroscedasticity, and outliers [14], the data are transformed applying the natural log (ln) to the data values to make the distribution closer to a normal distribution [14, 34]. 5 Training and testing the models Since AR is unbiased and does not lead to asymmetry [13], AR is selected as the accuracy criterion for evaluating the models. An AR is calculated as follows: It is calculated for each observation i, the maintenance effort of which is predicted. The aggregation of the AR over multiple observations (N) can be obtained by the mean of absolute residuals (MAR) as follows: In addition, the median of absolute residuals (MdAR) is calculated. The accuracy of a prediction model is inversely proportional to the MAR and MdAR. Regarding the validation method, LOOCV is used for training and testing the models [48]. After the LOOCV is applied, both MAR and MdAR obtained from ARs by data set of the test sample are calculated and then used to evaluate the SMEP accuracy of the models. The procedure followed for training and testing the SGB is formally described in Fig. 1 and applied as follows: Apply LOOCV by leaving out an element (i.e. a SISi) to be used as a test pattern If the independent variable (i.e. AFPi) presents duplicates in the training set, then Calculate the average of the dependent variable (i.e. Efforti) values of the patterns which have the same independent variable as the test pattern. Store the average value. Eliminate the repeated values in the training set. Average the patterns that have the same independent variable value in order to have a single pattern with that independent variable . Construct an ensemble using SGB. Predict the value of the test pattern. Fig. 1Open in figure viewerPowerPoint Algorithm for SMEP applying SGB If the process in step 2 was carried out, then return the average between the predicted value and the stored average. Otherwise, return the predicted value. 6 Results Table 2 shows the MAR and MdAR by data set obtained from the SLR, the MLP (whose prediction accuracy was reported as the best one among three neural networks [13]), the polynomial kernel ɛ-SVR (PKɛ-SVR) (the best one among eight types of SVR [14]), DT [14], and Aru [14], as well as from the SGB. Table 2. MAR and MdAR obtained from models () DP LT RL Size Accuracy measure SLR [13, 14] MLP [13] PKɛ-SVR [14] DT [14] ARu [14] SGB MF 3GL 1 267 MAR 0.50 0.42 0.50 0.51 0.52 0.36 — — — — MdAR 0.42 0.34 0.43 0.46 0.48 0.31 MF 3GL 4 50 MAR 0.53 0.51 0.49 0.61 0.59 0.47 — — — — MdAR 0.44 0.43 0.39 0.53 0.46 0.44 MR 3GL 1 39 MAR 0.52 0.48 0.47 0.57 0.59 0.49 — — — — MdAR 0.42 0.37 0.35 0.45 0.52 0.48 MP 3GL 1 102 MAR 0.54 0.51 0.54 0.60 0.58 0.47 — — — — MdAR 0.46 0.43 0.44 0.51 0.47 0.38 MP 4GL 1 65 MAR 0.48 0.47 0.49 0.57 0.54 0.45 — — — — MdAR 0.36 0.34 0.37 0.55 0.41 0.37 A selection of a suitable statistical test for comparing the two sets of ARs is based on the number of data sets to be compared, data dependence, as well as data normality [49]. In this study, data are dependent because those two models applied at the time are applied to each project. Therefore, it is needed to obtain a new set of n differences (n = data set size) between the two sets of ARs to determine if the mean of this data set is equal or greater than zero with statistically significance difference. If a set of differences resulted normally distributed after applying the four normality statistical tests (i.e. Shapiro–Wilk, Chi-square, skewness, and kurtosis) to the sets of differences obtained from Table 2, then a t-paired (based on MAR) statistical test was used to compare the prediction accuracy between two models, otherwise, a Wilcoxon (based on MdAR) test was applied [50]. Table 3 shows the p-values by comparing the SGB accuracy with each model of Table 2. In accordance with Tables 2 and 3, the SGB has a better statistically significant difference with SLR, PKɛ-SVR, DT, and ARu models at 99% of confidence, and better than MLP at 95% of confidence when applied to the first data set of SISs. The SGB has a better statistically significant difference with SLR, MLP, PKɛ-SVR and ARu models at 95% of confidence, and better than DT at 99% of confidence when applied to the fourth data set of SISs. When the SGB was applied to the second and fifth data set, the SGB was statistically equal than SLR, MLP, and PKɛ-SVR at 99% of confidence, and better than DT and ARu at 90% (second set) and at 95% (fifth set) of confidence. Finally, the SGB was statistically equal with all of the models at 99% of confidence when applied to the third data set. Table 3. Statistically significant comparison (p-value) between SGB and models by data set DP LT RL Size SLR MLP PKɛ-SVR DT ARu MF 3GL 1 267 0.0000 0.0455 0.0000 0.0000 0.0000 MF 3GL 4 50 0.2953 0.6521 0.7160 0.0505 0.0776 MR 3GL 1 39 0.7766 0.8704 0.8093 0.3271 0.3273 MP 3GL 1 102 0.0334 0.0497 0.0454 0.0048 0.0112 MP 4GL 1 65 0.3318 0.5875 0.1642 0.0347 0.0398 7 Discussion Thousands of developed projects need SM, which refers to anomalies uncovered, operating environments changed, and additional user requirements put forward after the software product has been delivered [3]. The contribution of our study was to investigate whether the SMEP accuracy obtained with a SGB was better than those obtained by 14 models derived from four models when five data sets of SISs obtained from the ISBSG data set, were used for training and testing the models. The procedure our study took into account is the guidelines suggested for comparing prediction models [13]. After applying statistical tests, we can accept the following hypothesis derived from that formulated in the Introduction section of our study when AFP data is used as the independent variable When SIS developed on MF, coded in 3GL, and having as RL 1: H1a: the maintenance effort prediction accuracy for a SIS with a SGB is statistically better at 99% of confidence than the accuracy obtained with SLR, PKɛ-SVR, DT and ARu models. H1b: the maintenance effort prediction accuracy for a SIS with a SGB is statistically better at 95% of confidence than the accuracy obtained with a MLP model. When SIS developed on MP, coded in 3GL, and having as RL 1: H1c: the maintenance effort prediction accuracy for a SIS with a SGB is statistically better at 95% of confidence than the accuracy obtained with SLR, MLP, PKɛ-SVR, ARu models. H1d: the maintenance effort prediction accuracy for a SIS with a SGB is statistically better at 99% of confidence than the accuracy obtained with the DT model. When SIS developed on MP, coded in 4GL, and having as RL 1: H1e: the maintenance effort prediction accuracy for a SIS with a SGB is statistically better at 95% of confidence than the accuracy obtained with DT and ARu models. H0e: the maintenance effort prediction accuracy for a SIS with a SGB is statistically equal at 99% of confidence than the accuracy obtained with SLR, MLP, and PKɛ-SVR models. When SIS developed on MF, coded in 3GL, and having as RL 4: H1f: the maintenance effort prediction accuracy for a SIS with a SGB is statistically better at 90% of confidence than the accuracy obtained with DT and ARu models. H0f: the maintenance effort prediction accuracy for a SIS with a SGB is statistically equal at 99% of confidence than the accuracy obtained with SLR, MLP, and PKɛ-SVR models. When SIS developed on MR, coded in 3GL, having as RL 1: H0g: the maintenance effort prediction accuracy for a SIS with a SGB is statistically equal at 99% of confidence than the accuracy obtained with SLR, MLP, PKɛ-SVR, DT and ARu models. Based on these hypotheses, we can conclude that the SMEP with an SGB outperformed the prediction accuracy of SLR, MLP, PKɛ-SVR, DT and ARu models for SIS developed on mainframes, coded in third generation programming languages and whose effort recording includes the efforts of the development team. SLR, MLP, PKɛ-SVR, DT and ARu models for SIS developed on multi-platform, coded in third generation programming languages and whose effort recording includes the efforts of the development team. DT and ARu models for SIS developed on multi-platform, coded in fourth generation programming languages and whose effort recording includes the efforts of the development team. DT and ARu models for SIS developed on mainframes, coded in third generation programming languages and whose effort recording includes the efforts of the development team, its support personnel, the computer operations involvement, and end users.In addition, since SGB resulted statistically equal with SLR, MLP, PKɛ-SVR, DT and ARu, a SGB for SMEP could be an alternative when a SIS was developed under the following conditions: On mid-range, coded in third generation programming languages and whose effort recording includes the efforts of the development team. The SGB had better prediction accuracy than the other five models when applied to those larger data sets of our study (i.e. with 267 and 102 SISs), which allowed obtaining a better generalisation about the SGB. In comparing our research with previous ones, we only identified 16 studies regarding SMEP in the last 23 years, and we did not find an SGB as one of their prediction models. We could only compare our results with two previous studies where our data sets were used [13, 14]. Regarding limitations of this study, although the version of the ISBSG dataset that we used contained 5052 SISs, we could only use five data sets; i.e. we could not compare the accuracy of the SGB using other kinds of maintained SISs developed on other types of platforms or programing language generations. A second limitation is that the type of maintenance that ISBSG reports is an enhancement, therefore, we could not predict the effort for SISs whose type of maintenance had been corrective, adaptive or reengineering. Regarding SGB, since that it is a stochastic algorithm, it requires the generation of a random subset of the training data, which can vary for each problem. Additionally, other parameters such as the number of iterations and the number of trees are difficult to estimate and they depend on the data set. We could identify the following validation threats in our study: Internal: the selected projects were those ones having the highe