# A New Software Reliability Growth Model: Multigeneration Faults and a Power-Law Testing-Effort Function

Authors: Fan Li; Ze-Long Yi
Journal: Mathematical Problems in Engineering
 (2016)

---

## Abstract

Software reliability growth models (SRGMs) based on a nonhomogeneous Poisson process (NHPP) are widely used to describe the stochastic failure behavior and assess the reliability of software systems. For these models, the testing-effort effect and the fault interdependency play significant roles. Considering a power-law function of testing effort and the interdependency of multigeneration faults, we propose a modified SRGM to reconsider the reliability of open source software (OSS) systems and then to validate the model’s performance using several real-world data. Our empirical experiments show that the model well fits the failure data and presents a high-level prediction capability. We also formally examine the optimal policy of software release, considering both the testing cost and the reliability requirement. By conducting sensitivity analysis, we find that if the testing-effort effect or the fault interdependency was ignored, the best time to release software would be seriously delayed and more resources would be misplaced in testing the software.


---

## Body

## 1. Introduction

Software systems are thought to run successfully when they perform without any failure during a certain period of time. However, in practice, failures are frequently observed and a perfect operational process cannot always be guaranteed due to a variety of reasons. Software reliability, which is defined as the probability of failure-free operation of a software program for a specified time interval, is an essential consideration in the processes of software development and a critical factor in measurement of software quality. In fact, accurately modeling the software reliability growth process and predicting its possible trend are of high values for software developers. For these reasons, a bunch of software reliability growth models (SRGMs), most of which are based on a nonhomogeneous Poisson process (NHPP), have been constructed [1–5]. Unlike closed-source software, open source software’s (OSS) knowledge base is not bounded to a particular organization. As a result, reliability measurement is important and complicated for OSS, such as Linux operating systems, Mozilla browser, and Apache web server. Due to their unique properties, a parallel of SRGMs has been purposefully developed to quantify the stochastic failure behavior of OSS [6, 7].One key factor of software reliability growth modeling is the testing-effort effect [8–14]. Huang and Kuo [15] incorporated a logistic testing-effort function into software reliability modeling and showed that it provides powerful prediction capability based on real failure data. Huang [16] considered a generalized logistic testing-effort function and change-point parameters in software reliability modeling and some practical data were employed to fit the new models. Ahmad et al. [17] incorporated the exponentiated Weibull testing-effort function into inflection S-shaped software reliability growth models. Peng et al. [18] established a flexible and general software reliability model considering both testing-effort function and imperfect debugging.In these models, assuming independence among successive software runs may not be appropriate [19–21]. For instance, Goševa-Popstojanova and Trivedi [22] introduced a software reliability model based on Markov renewal processes that covers interdependence among successive software runs. The Compound-Poisson software reliability model presented by Sahinoglu [23] considered multiple failures that occur simultaneously. Singh et al. [24] proposed new SRGMs that assume the presence of two types of faults in the software: leading and dependent. Leading faults can be removed when a failure is observed, but dependent faults are masked by leading faults and can be removed only after the corresponding leading faults are removed.We also notice that a bundle of prior papers have examined the problems of optimal release time under different criteria. One stream studies this issue by minimizing the total cost subject to a constraint on software reliability. For example, Huang and Lin [25] provided a series of theoretical findings regarding when a software version should be released and when fault dependency and debugging time lag are considered. In addition, Ahmad et al. [26] analyzed the optimal release timing in an SRGM with Burr-type X testing-effort functions. Another stream adopts the multiattribute utility theory (MAUT) to decide the release time. For instance, Singh et al. [27] built an SRGM based on the Yamada two-stage model and focused on two different attributes to investigate the release time. Pachauri et al. [28] studied an SRGM incorporating a generalized modified Weibull testing-effort function in an imperfect debugging environment and the optimal release policy is examined using both genetic algorithm (GA) and MAUT.In this work, we propose a new model that incorporates a power-law testing-effort function and three generations of interdependent errors. Specifically, the detection process for the first-generation errors is independent, and the detection rate is proportional to the instantaneous testing-effort expenditure and the mean of remaining first-generation errors in the system. By contrast, the detection process of the second-generation errors depends on the first generation and the detection rate is proportional to not only the instantaneous testing-effort expenditure and the mean of remaining second-generation errors but also the ratio of removed first-generation errors. Similarly, the detection process of the third-generation errors relies on the second-generation errors and the detection rate is proportional to the instantaneous testing-effort expenditure, the mean of remaining third-generation errors, and the ratio of removed second-generation errors.Model parameters are estimated by the nonlinear least square estimation method, which is realized by minimizing the sum of squares of the deviations between estimated values and true observations. Numerical experiments are carried out based on three versions of Apache. The estimation results show that the proposed model well fits the real observations and performs better than the traditional Goel-Okumoto model and the Yamada delayed S-shaped model. Comprehensive comparisons in terms of prediction capabilities among various models are conducted to reveal that our model can predict the error occurrence process more accurately than the benchmarks. We also analyze the optimal release policy and the minimal cost for the software development managers, based on which insightful suggestions are provided. Besides, we conduct sensitivity analysis and find that resources would be misplaced if either the testing-effort effect or the error interdependency is overlooked by the software testing team. Unlike the benchmarks, our results suggest that the strategy of an earlier release time can substantially reduce the cost.


## 2. Software Reliability Growth Modeling

### 2.1. Basic Assumptions

To describe the stochastic failure behavior of software systems, we define{N(t),t≥0} as a counting process that represents the cumulative number of failures by time t. We use the mean value function (MVF) mt to denote the expected cumulative number of detected failures in the time period (0,t]. That is,(1)mt=ENt=∫0tλt′dt′,where E[·] means the expectation operator and λ(t′) is called failure intensity function which indicates the instantaneous fault-detection rate at time t′. Specifically, for an NHPP-based SRGM, it is assumed that N(t) follows a Poisson distribution with a mean value function m(t). That is, the model can be mathematically characterized as(2)PrNt=n=mtn×e-mtn!=Poimn,mt,n=0,1,2…,where Poim(n,m(t)) is the Poisson probability mass function (pmf) with mean m(t) [29, 30]. Generally, one can obtain different NHPP models by taking different mean value functions.In this paper, to characterize the stochastic behavior of the software failure process and obtain the expression for the mean value function, we make the following assumptions:(i)
The software system is subject to failures at random times caused by errors remaining in the system.(ii)
The error detection phenomenon in software testing is modeled by a nonhomogeneous Poisson process.(iii)
Each time a failure occurs, the error causing it is immediately removed and no new errors are introduced.(iv)
The faults existing in the software are of three generations(G1,G2,andG3) and each generation is modeled by a different growth curve.(v)
The mean number ofG1 errors detected in the time interval (t,t+Δt) is proportional to the instantaneous testing-effort expenditure and the mean number of remaining G1 errors in the system.(vi)
The mean number ofG2 errors detected in the time interval (t,t+Δt) is proportional to the instantaneous testing-effort expenditure, the mean number of remaining G2 errors in the system, and the ratio of removed G1 errors.(vii)
The mean number ofG3 errors detected in the time interval (t,t+Δt) is proportional to the instantaneous testing-effort expenditure, the mean number of remaining G3 errors in the system, and the ratio of removed G2 errors.(viii)
Fault-detection/removal rate is a power-law function of testing time for all three generations of errors.The first four assumptions in the above list are the building blocks in most prior software reliability growth models and the others are specific in our model. To facilitate our later analysis, we summarize the notations used throughout this paper in Notations section.


### 2.2. Proposed Software Reliability Growth Model

In general, an implemented software system is tested to detect and correct software errors in the development process. During the testing phase, software errors remaining in the system are discovered and corrected by test personnel. Based on the aforementioned assumptions, the detection rate must be proportional to the instantaneous testing-effort expenditure. Particularly, unlike traditional closed-source software, in open source software, instructions are executed by various concurrent users and each release of open source software can attract an increasing number of volunteers in the software development and/or code modification procedure. Hence, it is reasonable to assume the detection/correction rate as a power-law function of testing time [19, 24]. That is, the instantaneous testing-effort expenditure at time t, w(t), takes the following form:(3)wt=btk,where b and k are parameters to be estimated by real-world data sets. As such, we can quickly obtain the cumulative amount of testing-effort expenditures by time t; that is,(4)Wt=∫0twt′dt′=btk+1k+1.As stated above, it is essential to consider fault interdependency when modeling software reliability growth processes. Specifically, in this work we propose three generations of interdependent errors. That is, the detection of the second-generation errors relies on the first generation, and the detection of the third-generation errors depends on the second generation. Moreover, the second-generation errors are detectable only if the first-generation errors have been removed, and the third-generation errors are detectable only if the second-generation errors have been removed. The flow chart is shown in Figure1 to illustrate the error detecting process in a software system.Figure 1
A flow chart for the error testing process when multigeneration faults exist in a software system.Next, we sequentially examine the mean value function for each generation of errors:(i) For the first-generation errors in the software system, the number of detected errors is proportional to the number of remaining errors and the instantaneous testing-effort expenditure. Thus, the mean value function forG1 errors satisfies the following differential equation:(5)dm1tdt=wta1-m1t,where a1 is the initial content of G1 errors in the system and m1t represents the expected number of G1 errors detected in time (0,t]. Specifically, m1t is assumed to be a bounded nondecreasing function of t with the initial condition m10=0. Solving the above differential equation yields(6)m1t=a11-e-btk+1/k+1.(ii) The mean value function forG2 errors satisfies the following differential equation:(7)dm2tdt=wta2-m2tm1ta1,where a2 is the initial content of G2 errors in the system and m2(t) stands for the expected number of G2 errors detected in time (0,t]. Specifically, m2(t) is also assumed to be a bounded nondecreasing function of t with the initial condition m2(0)=0. Solving the above differential equation yields(8)m2t=a21-e-btk+1/k+1-e-btk+1/k+1+1.(iii) Similarly, the mean value function forG3 errors satisfies the following differential equation:(9)dm3tdt=wta3-m3tm2ta2,where a3 is the initial content of G3 errors in the system and m3(t) means the expected number of G3 errors detected in time (0,t]. Specifically, m3(t) is assumed to be a bounded nondecreasing function of t with the initial condition m3(0)=0. Solving the above differential equation leads to(10)m3t=a31-e-btk+1/k+1+e1-e-btk+1/k+1-1.After this, we formulate the overall mean value functionm(t) that consists of mean value functions for three generations of errors. Assume that a=a1+a2+a3 is the expected total number of faults that will be eventually detected and p1, p2, and p3 are corresponding ratios for each generation of errors at the beginning of testing. By noting that a1=ap1, a2=ap2, a3=ap3, and p1+p2+p3=1, we have(11)mt=m1t+m2t+m3t=a11-e-btk+1/k+1+a21-e-btk+1/k+1-e-btk+1/k+1+1+a31-e-btk+1/k+1+e1-e-btk+1/k+1-1=a1-p1e-btk+1/k+1-p2e-btk+1/k+1-e-btk+1/k+1+1-p3e-btk+1/k+1+e1-e-btk+1/k+1-1=a1-p1e-btk+1/k+1-p2e-btk+1/k+1-e-btk+1/k+1+1-1-p1-p2e-btk+1/k+1+e1-e-btk+1/k+1-1.As such, m(t) is a bounded nondecreasing function of t, with m(0)=0 and m(∞)=a.Then, based on the aforesaid mean value function, the failure intensity function for the proposed model, with testing-effort, can be explicitly given as(12)λt=dmtdt=wta1e-btk+1/k+1+a2e-btk+1/k+1-e-btk+1/k+1+11-e-btk+1/k+1+a3e-btk+1/k+1+e1-e-btk+1/k+1-11-e-btk+1/k+1-e-btk+1/k+1+1=abtkp1e-btk+1/k+1+p2e-btk+1/k+1-e-btk+1/k+1+11-e-btk+1/k+1+1-p1-p2e-btk+1/k+1+e1-e-btk+1/k+1-11-e-btk+1/k+1-e-btk+1/k+1+1.We further note that whenk=0, p1=1, and p2=p3=0, the original Goel-Okumoto model can be realized [31]; that is,(13)mt=a1-e-bt,which is one of our benchmarks for the comparative analysis. In this case, errors are assumed to be independent and the testing-effort function keeps constant, that is, b. Besides, the mean value function m(t) is a concave function with respect to t.Moreover, whenk=0, we have w(t)=b; namely, the instantaneous testing-effort expenditure at time t reduces to a constant. In this case, we have(14)mt=a1-p1e-bt-p2e-bt-e-bt+1-1-p1-p2e-bt+e1-e-bt-1,and we call it No-k model for brevity.In addition, whenp1=1 and p2=p3=0, the error interdependency is ignored and only one generation of errors exists in the software system. As a result, we have(15)mt=a1-e-btk+1/k+1.We call it Only-G1 model for exposition.As another benchmark, hereby we introduce an S-shaped model, which incorporates an increasing detection rate function and well explains “learning effect” of the testing team. Essentially, the mean value function is stated as follows:(16)mt=a1-1+bte-bt.It is also called Yamada delayed S-shaped model [32, 33] in literature. For this model, the detection rate first increases and then decreases with t.


## 2.1. Basic Assumptions

To describe the stochastic failure behavior of software systems, we define{N(t),t≥0} as a counting process that represents the cumulative number of failures by time t. We use the mean value function (MVF) mt to denote the expected cumulative number of detected failures in the time period (0,t]. That is,(1)mt=ENt=∫0tλt′dt′,where E[·] means the expectation operator and λ(t′) is called failure intensity function which indicates the instantaneous fault-detection rate at time t′. Specifically, for an NHPP-based SRGM, it is assumed that N(t) follows a Poisson distribution with a mean value function m(t). That is, the model can be mathematically characterized as(2)PrNt=n=mtn×e-mtn!=Poimn,mt,n=0,1,2…,where Poim(n,m(t)) is the Poisson probability mass function (pmf) with mean m(t) [29, 30]. Generally, one can obtain different NHPP models by taking different mean value functions.In this paper, to characterize the stochastic behavior of the software failure process and obtain the expression for the mean value function, we make the following assumptions:(i)
The software system is subject to failures at random times caused by errors remaining in the system.(ii)
The error detection phenomenon in software testing is modeled by a nonhomogeneous Poisson process.(iii)
Each time a failure occurs, the error causing it is immediately removed and no new errors are introduced.(iv)
The faults existing in the software are of three generations(G1,G2,andG3) and each generation is modeled by a different growth curve.(v)
The mean number ofG1 errors detected in the time interval (t,t+Δt) is proportional to the instantaneous testing-effort expenditure and the mean number of remaining G1 errors in the system.(vi)
The mean number ofG2 errors detected in the time interval (t,t+Δt) is proportional to the instantaneous testing-effort expenditure, the mean number of remaining G2 errors in the system, and the ratio of removed G1 errors.(vii)
The mean number ofG3 errors detected in the time interval (t,t+Δt) is proportional to the instantaneous testing-effort expenditure, the mean number of remaining G3 errors in the system, and the ratio of removed G2 errors.(viii)
Fault-detection/removal rate is a power-law function of testing time for all three generations of errors.The first four assumptions in the above list are the building blocks in most prior software reliability growth models and the others are specific in our model. To facilitate our later analysis, we summarize the notations used throughout this paper in Notations section.


## 2.2. Proposed Software Reliability Growth Model

In general, an implemented software system is tested to detect and correct software errors in the development process. During the testing phase, software errors remaining in the system are discovered and corrected by test personnel. Based on the aforementioned assumptions, the detection rate must be proportional to the instantaneous testing-effort expenditure. Particularly, unlike traditional closed-source software, in open source software, instructions are executed by various concurrent users and each release of open source software can attract an increasing number of volunteers in the software development and/or code modification procedure. Hence, it is reasonable to assume the detection/correction rate as a power-law function of testing time [19, 24]. That is, the instantaneous testing-effort expenditure at time t, w(t), takes the following form:(3)wt=btk,where b and k are parameters to be estimated by real-world data sets. As such, we can quickly obtain the cumulative amount of testing-effort expenditures by time t; that is,(4)Wt=∫0twt′dt′=btk+1k+1.As stated above, it is essential to consider fault interdependency when modeling software reliability growth processes. Specifically, in this work we propose three generations of interdependent errors. That is, the detection of the second-generation errors relies on the first generation, and the detection of the third-generation errors depends on the second generation. Moreover, the second-generation errors are detectable only if the first-generation errors have been removed, and the third-generation errors are detectable only if the second-generation errors have been removed. The flow chart is shown in Figure1 to illustrate the error detecting process in a software system.Figure 1
A flow chart for the error testing process when multigeneration faults exist in a software system.Next, we sequentially examine the mean value function for each generation of errors:(i) For the first-generation errors in the software system, the number of detected errors is proportional to the number of remaining errors and the instantaneous testing-effort expenditure. Thus, the mean value function forG1 errors satisfies the following differential equation:(5)dm1tdt=wta1-m1t,where a1 is the initial content of G1 errors in the system and m1t represents the expected number of G1 errors detected in time (0,t]. Specifically, m1t is assumed to be a bounded nondecreasing function of t with the initial condition m10=0. Solving the above differential equation yields(6)m1t=a11-e-btk+1/k+1.(ii) The mean value function forG2 errors satisfies the following differential equation:(7)dm2tdt=wta2-m2tm1ta1,where a2 is the initial content of G2 errors in the system and m2(t) stands for the expected number of G2 errors detected in time (0,t]. Specifically, m2(t) is also assumed to be a bounded nondecreasing function of t with the initial condition m2(0)=0. Solving the above differential equation yields(8)m2t=a21-e-btk+1/k+1-e-btk+1/k+1+1.(iii) Similarly, the mean value function forG3 errors satisfies the following differential equation:(9)dm3tdt=wta3-m3tm2ta2,where a3 is the initial content of G3 errors in the system and m3(t) means the expected number of G3 errors detected in time (0,t]. Specifically, m3(t) is assumed to be a bounded nondecreasing function of t with the initial condition m3(0)=0. Solving the above differential equation leads to(10)m3t=a31-e-btk+1/k+1+e1-e-btk+1/k+1-1.After this, we formulate the overall mean value functionm(t) that consists of mean value functions for three generations of errors. Assume that a=a1+a2+a3 is the expected total number of faults that will be eventually detected and p1, p2, and p3 are corresponding ratios for each generation of errors at the beginning of testing. By noting that a1=ap1, a2=ap2, a3=ap3, and p1+p2+p3=1, we have(11)mt=m1t+m2t+m3t=a11-e-btk+1/k+1+a21-e-btk+1/k+1-e-btk+1/k+1+1+a31-e-btk+1/k+1+e1-e-btk+1/k+1-1=a1-p1e-btk+1/k+1-p2e-btk+1/k+1-e-btk+1/k+1+1-p3e-btk+1/k+1+e1-e-btk+1/k+1-1=a1-p1e-btk+1/k+1-p2e-btk+1/k+1-e-btk+1/k+1+1-1-p1-p2e-btk+1/k+1+e1-e-btk+1/k+1-1.As such, m(t) is a bounded nondecreasing function of t, with m(0)=0 and m(∞)=a.Then, based on the aforesaid mean value function, the failure intensity function for the proposed model, with testing-effort, can be explicitly given as(12)λt=dmtdt=wta1e-btk+1/k+1+a2e-btk+1/k+1-e-btk+1/k+1+11-e-btk+1/k+1+a3e-btk+1/k+1+e1-e-btk+1/k+1-11-e-btk+1/k+1-e-btk+1/k+1+1=abtkp1e-btk+1/k+1+p2e-btk+1/k+1-e-btk+1/k+1+11-e-btk+1/k+1+1-p1-p2e-btk+1/k+1+e1-e-btk+1/k+1-11-e-btk+1/k+1-e-btk+1/k+1+1.We further note that whenk=0, p1=1, and p2=p3=0, the original Goel-Okumoto model can be realized [31]; that is,(13)mt=a1-e-bt,which is one of our benchmarks for the comparative analysis. In this case, errors are assumed to be independent and the testing-effort function keeps constant, that is, b. Besides, the mean value function m(t) is a concave function with respect to t.Moreover, whenk=0, we have w(t)=b; namely, the instantaneous testing-effort expenditure at time t reduces to a constant. In this case, we have(14)mt=a1-p1e-bt-p2e-bt-e-bt+1-1-p1-p2e-bt+e1-e-bt-1,and we call it No-k model for brevity.In addition, whenp1=1 and p2=p3=0, the error interdependency is ignored and only one generation of errors exists in the software system. As a result, we have(15)mt=a1-e-btk+1/k+1.We call it Only-G1 model for exposition.As another benchmark, hereby we introduce an S-shaped model, which incorporates an increasing detection rate function and well explains “learning effect” of the testing team. Essentially, the mean value function is stated as follows:(16)mt=a1-1+bte-bt.It is also called Yamada delayed S-shaped model [32, 33] in literature. For this model, the detection rate first increases and then decreases with t.


## 3. Estimation and Analysis

### 3.1. Data Set

To compare the proposed model against traditional models for reliability assessment, numerical examples are provided based on real-world data. Specifically, the well-known Apache project, the most used web server software that is developed and maintained by an open community of developers, is selected in this paper for illustration. This project has large and well-organized communities: a great number of developers have the right to update and change files freely. In particular, testing data about three versions of Apache, 2.0.35/2.0.36/2.0.39, are used. The failure data is presented in Table1 [8], including the time and the number of errors detected for each version. Specifically, testing for Apache 2.0.35 ranges from day 1 to day 43 and in total 74 errors are found; failure data for Apache 2.0.36 are collected from day 1 to day 103 and 50 errors are detected; Apache 2.0.39 is tested over a 164-day period and in the end 58 errors are discovered by the testing team.Table 1
Detected faults (days from release and number of errors) for three versions of Apache, 2.0.35/2.0.36/2.0.39. Note that some failure data is not shown due to page limit. For example, as no faults are detected on the 42nd day in Apache 2.0.35, no failure data is shown on this day.
Apache 2.0.35 Apache 2.0.36 Apache 2.0.39 Days from release Detected errors Days from release Detected errors Days from release Detected errors 1 5 1 2 1 1 2 5 2 5 2 2 3 4 3 1 3 2 4 1 4 1 4 3 5 2 5 1 5 3 6 4 7 2 7 2 7 6 8 1 8 1 8 2 9 1 9 1 10 1 10 3 10 1 11 8 12 2 11 1 12 5 13 1 15 3 13 2 15 2 16 2 14 2 17 1 17 3 15 1 18 2 18 1 17 2 21 1 19 1 18 3 25 1 22 3 19 4 27 1 23 1 20 1 29 2 24 1 21 4 30 2 25 2 23 2 31 3 26 1 24 1 32 1 28 1 25 1 33 3 29 1 26 2 34 1 30 2 27 1 35 3 31 1 28 2 38 3 32 1 31 1 40 1 35 3 34 1 43 1 38 1 43 1 44 1 39 1 103 1 42 1 43 1 49 3 50 1 51 1 57 1 66 1 70 1 81 1 164 1


### 3.2. Estimation Method

The nonlinear least square estimation (NLSE) method is a widely used estimation technique, especially for small/medium sized data sets [34]. We will use this technique to estimate parameters in models raised in this paper. More specifically, estimation is done by minimizing the sum of squared residuals, which are the squares of the differences between estimated values and true observations. The objective function for the minimization problem is stated as follows:(17)χ2=∑j=1Kmtj-mj2,where mj is the jth observation, K is the total number of observations, tj is the time in which the jth error occurs, and m(tj) is the corresponding estimated/theoretical value. Specifically, in this paper, m(tj) can be from (11), (13), (14), (15), or (16). They correspond to our proposed model, the Goel-Okumoto model, the No-k model, the Only-G1 model, and the Yamada delayed S-shaped, respectively.Differentiatingχ2 with respect to each unknown parameter and setting the partial derivatives to zero, we can obtain the estimated values [34, 35]. Specifically, for our proposed model, we have(18)∂χ2∂a=∂χ2∂b=∂χ2∂p1=∂χ2∂p2=∂χ2∂k=0.Using numerical methods to solve the set of simultaneous equations above yields the estimation for unknown parameters in our proposed model.


### 3.3. Metrics for Model Comparison

With estimated parameters, we can conduct comparative analysis among models based on the following metrics:(1) Goodness of Fit (R2): the value of R2 determines how well the curve fits the real data [19, 36]. It is defined as follows:(19)R2=1-∑j=1Kmtj-mj2∑j=1Kmj-∑l=1Kml/K2.It measures the percentage of the total variance about the mean accounted for the fitted curve. It ranges from 0 to 1. The larger R2 is, the better the model fits data. In this work, the model is considered to provide a good fit if R2 is above 0.95.(2) Mean of Square Error (MSE): the value of MSE measures the average of squares of the differences between the predicted values and the actual values [9]. It is defined as follows:(20)MSE=1K∑j=1Kmtj-mj2.Generally, the closer to zero the MSE value is, the better the model fits the data.(3) Theil Statistic (TS): the value of TS indicates the average percentage deviation with respect to the actual failure data during the selected time period [37]. It is defined as follows:(21)TS=∑j=1Kmtj-mj2∑j=1Kyj2.A smaller TS value means a smaller deviation and a better prediction capability. In this work, the model is considered to provide a high-level prediction if the TS value is below 10%.(4) Relative Error (RE): the value of RE measures the relative difference between the predicted value and the real observations [25, 38]. The definition can be stated as follows:(22)RE=mtq-mqmq.Specifically, assuming that we have observed mq failures by the end of test at time tq, the failure data up to time te (te≤tq) will be used to estimate parameters in the models. Substituting the estimated parameters in the mean value function yields the estimation of the number of failures m(tq) by tq. The estimation is then compared with the actual number mq. Such procedure is repeated for various values of te. Positive values of error indicate overestimation and negative values indicate underestimation.


### 3.4. Estimation Results and Model Comparison

The parameter estimation can help characterize properties of theoretical models, determine the number of errors that have already been detected/corrected in the testing process, and predict the number of errors that will be eventually encountered by users. Specifically, by minimizing the objective function in (17), we can obtain the estimated results for each model.(i) For Apache 2.0.35, the estimated results are presented in Table2, with a^=74.8102, b^=0.0432, p1^=0.5905, p2^=0.2838, and k^=0.3554 for our proposed model. Then, the ratio of G1 errors to all types of errors is approximately 59%, the ratio of G2 errors is around 28%, and the leftover are G3 errors. Note that the errors consist of not only independent errors (G1), although G2 and G3 account much less than G1. Through incorporating such interdependency among generations of errors, we are able to strengthen the understanding towards composition of software system. Besides, we find that the estimated value of k is obviously larger than 0. Hence, not considering the effect of testing effort in the analysis of the reliability growth process may lead to significantly biased results.Table 2
Optimal estimated values: Apache 2.0.35.
a ^ b ^ p 1 ^ p 2 ^ p 3 ^ k ^ R 2 MSE TS Goel-Okumoto model 84.6047 0.0564 — — — — 0.9873 5.7624 0.0417 Yamada delayed S-shaped model 74.2741 0.1539 — — — — 0.9830 7.7033 0.0482 No-k model 76.4154 0.1087 0.1593 0.7293 0.111 4 ∗ — 0.9947 2.3912 0.0269 Only-G1 model 77.1477 0.0438 — — — 0.2323 0.9925 3.4129 0.0321 Proposed model 74.8102 0.0432 0.5905 0.2838 0.125 8 ∗ 0.3554 0.9954 2.0979 0.0252 ∗Calculated by p3^=1-p1^-p2^.Illustrations for the theoretical values versus their observations are shown in Figure2. We find that the theoretical values are highly consistent with the real-world observations. We can further examine the failure intensity function λ(t). Figure 3 shows that the failure intensity function is a single-peaked curve; that is, the error detection rate first increases and then decreases with time. It in turn implies that our proposed model falls into the S-shaped category for the SRGMs [34].Figure 2
Predicted values and real-world observations for Apache 2.0.35. Solid line: predicted values; dot: observations.Figure 3
Failure intensity function with respect to time, with estimated values based on Apache 2.0.35.We observe that our proposed model well fits the real-world software failure data. This can be seen from the high value ofR2, that is, 0.9954, which is very close to 1. By comparing with the Goel-Okumoto model, the Yamada delayed S-shaped model, the No-k model, and the Only-G1 model, R2 values are the highest in our proposed model. This conclusion can also be drawn by comparing MSE values among the three models; our proposed model provides the lowest MSE values (2.0979). From this aspect, the model in our paper best fits the observed data about Apache 2.0.35.Furthermore, the prediction capability in our model is very high. The low TS value for Apache 2.0.35, 0.0252, is much lower than 10% and well confirms a high prediction capability for our model. Compared with the benchmarks, our model can provide significantly lower TS values and higher predication capacity to the failure of Apache system. Lastly, the relative error in prediction using this data set is computed and results are plotted in Figure4. We observe that the relative error approaches 0 as te approaches tq and the relative error curve is usually within ±5%.Figure 4
Predicted values and real-world observations for Apache 2.0.36. Solid line: predicted values; dot: observations.(ii) For Apache 2.0.36, the estimated results are presented in Table3, with a^=49.4122, b^=0.0045, p1^=0.4031, p2^=0.2855, and k^=0.8967 for our proposed model. Three generations of errors account for 40%, 29%, and 31%, respectively. Obviously, G2 and G3 significantly contribute to the error composition, which counts for 60% in total. We also find that the estimated value of k is much larger than 0 implying that the testing-effort effect is necessary.Table 3
Optimal estimated values: Apache 2.0.36.
a ^ b ^ p 1 ^ p 2 ^ p 3 ^ k ^ R 2 MSE TS Goel-Okumoto model 52.3158 0.0393 — — — — 0.9526 8.8417 0.0707 Yamada delayed S-shaped model 49.8964 0.0896 — — — — 0.9550 8.3927 0.0689 No-k model 50.4507 0.0669 0.0000 0.8065 0.193 5 ∗ — 0.9706 5.4946 0.0557 Only-G1 model 50.1765 0.0196 — — — 0.3207 0.9643 6.6635 0.0614 Proposed model 49.4122 0.0045 0.4031 0.2855 0.311 4 ∗ 0.8967 0.9806 3.6163 0.0452 ∗Calculated by p3^=1-p1^-p2^.Then, we can obtain theoretical values in our proposed model. Illustrations for the theoretical values versus their observations are presented in Figure5, which shows that the theoretical values are consistent with the real-world data. In Figure 6, we present the failure intensity function λ(t) based on the estimated values. Similar to Apache 2.3.35, the hump-shaped curve shows that this function possesses an increasing-then-decreasing trend.Figure 5
Failure intensity function with respect to time, with estimated values based on Apache 2.0.36.Figure 6
Predicted values and real-world observations for Apache 2.0.39. Solid line: predicted values; dot: observations.We find that our proposed model also well fits the real-world software failure data for Apache 2.0.36. This can be seen from the high value ofR2, 0.9806. It is also much higher than those in the Goel-Okumoto model, the Yamada delayed S-shaped model, the No-k model, and the Only-G1 model. By comparing MSE values among the models, our proposed model provides the lowest MSE value (3.6163). From this aspect, the proposed model best fits the observed data about Apache 2.0.36.Furthermore, the TS value for Apache 2.0.36 is as low as 0.0452, which is much lower than 10% and well confirms a high prediction capability of our model. Compared with the benchmarks, our model shows a lower value of TS and a higher level of predication capability. Lastly, the relative error in prediction for this data set is computed, as shown in Figure7. We observe that the relative error approaches 0 as te approaches tq and the curve is usually within ±5%.Figure 7
Failure intensity function with respect to time, with estimated values based on Apache 2.0.39.(iii) For Apache 2.0.39, the estimated results are presented in Table4, with a^=57.1499, b^=0.0300, p1^=0.5299, p2^=0.3820, and k^=0.1940 for our proposed model. The ratios of three generations of errors become about 53%, 38%, and 9%, respectively. Then, the ratio for G1 errors is approximately 59%, the ratio for G2 errors is around 28%, and the leftover are G3 errors. We also find that the estimated value of k is significantly greater than 0. Thus, ignoring the effect of testing effort in the analysis of the software reliability growth curve would lead to significant bias.Table 4
Optimal estimated values: Apache 2.0.39.
a ^ b ^ p 1 ^ p 2 ^ p 3 ^ k ^ R 2 MSE TS Goel-Okumoto model 58.3827 0.0367 — — — — 0.9871 2.5670 0.0316 Yamada delayed S-shaped model 56.8974 0.0805 — — — — 0.9879 2.4045 0.0306 No-k model 57.3279 0.0577 0.2414 0.6741 0.084 5 ∗ — 0.9969 0.6133 0.0154 Only-G1 model 57.3266 0.0204 — — — 0.2453 0.9957 0.8562 0.0182 Proposed model 57.1499 0.0300 0.5299 0.3820 0.088 1 ∗ 0.1940 0.9973 0.5314 0.0144 ∗Calculated by p3^=1-p1^-p2^.Comparison between the fitted curve and the observed failure data is illustrated in Figure8, which indicates that the theoretical values and the real-world observations are highly consistent. In Figure 9, we present the failure intensity function based on the estimated values. Obviously, it is also a single-peaked curve: the error detection rate first increases and then decreases with time.Figure 8
Relative error for Apache 2.0.35.Figure 9
Relative error for Apache 2.0.36.The high value ofR2, 0.9937, suggests that our proposed model well fits the real-world software failure data for this version. By comparing with the Goel-Okumoto model, the Yamada delayed S-shaped model, the No-k model, and the Only-G1 model, R2 values are the highest in our proposed model. This conclusion can also be drawn by comparing MSE values among the models; our proposed model provides the lowest MSE value (0.5314). Therefore, our proposed model best fits the observed data about Apache 2.0.39.Furthermore, the TS value for Apache 2.0.39 is 0.0144. It is much lower than 10% and well confirms a high prediction capability for our model. Compared with the benchmarks, TS values in our model are significantly lower, and prediction capacity of the failure occurrence is also much greater. Lastly, the relative error in prediction for this data set is computed and results are plotted in Figure10. We observe that the relative error approaches 0 as te approaches tq and the error curve is usually within ±5%.Figure 10
Relative error for Apache 2.0.39.In summary, all of three tests using different versions of Apache failure data sets suggest that our proposed model well fits the real data and presents better performance than the benchmarks in error occurrence prediction. Besides, both the testing effort and the error interdependency effects play an important role in the parameter estimation.


## 3.1. Data Set

To compare the proposed model against traditional models for reliability assessment, numerical examples are provided based on real-world data. Specifically, the well-known Apache project, the most used web server software that is developed and maintained by an open community of developers, is selected in this paper for illustration. This project has large and well-organized communities: a great number of developers have the right to update and change files freely. In particular, testing data about three versions of Apache, 2.0.35/2.0.36/2.0.39, are used. The failure data is presented in Table1 [8], including the time and the number of errors detected for each version. Specifically, testing for Apache 2.0.35 ranges from day 1 to day 43 and in total 74 errors are found; failure data for Apache 2.0.36 are collected from day 1 to day 103 and 50 errors are detected; Apache 2.0.39 is tested over a 164-day period and in the end 58 errors are discovered by the testing team.Table 1
Detected faults (days from release and number of errors) for three versions of Apache, 2.0.35/2.0.36/2.0.39. Note that some failure data is not shown due to page limit. For example, as no faults are detected on the 42nd day in Apache 2.0.35, no failure data is shown on this day.
Apache 2.0.35 Apache 2.0.36 Apache 2.0.39 Days from release Detected errors Days from release Detected errors Days from release Detected errors 1 5 1 2 1 1 2 5 2 5 2 2 3 4 3 1 3 2 4 1 4 1 4 3 5 2 5 1 5 3 6 4 7 2 7 2 7 6 8 1 8 1 8 2 9 1 9 1 10 1 10 3 10 1 11 8 12 2 11 1 12 5 13 1 15 3 13 2 15 2 16 2 14 2 17 1 17 3 15 1 18 2 18 1 17 2 21 1 19 1 18 3 25 1 22 3 19 4 27 1 23 1 20 1 29 2 24 1 21 4 30 2 25 2 23 2 31 3 26 1 24 1 32 1 28 1 25 1 33 3 29 1 26 2 34 1 30 2 27 1 35 3 31 1 28 2 38 3 32 1 31 1 40 1 35 3 34 1 43 1 38 1 43 1 44 1 39 1 103 1 42 1 43 1 49 3 50 1 51 1 57 1 66 1 70 1 81 1 164 1


## 3.2. Estimation Method

The nonlinear least square estimation (NLSE) method is a widely used estimation technique, especially for small/medium sized data sets [34]. We will use this technique to estimate parameters in models raised in this paper. More specifically, estimation is done by minimizing the sum of squared residuals, which are the squares of the differences between estimated values and true observations. The objective function for the minimization problem is stated as follows:(17)χ2=∑j=1Kmtj-mj2,where mj is the jth observation, K is the total number of observations, tj is the time in which the jth error occurs, and m(tj) is the corresponding estimated/theoretical value. Specifically, in this paper, m(tj) can be from (11), (13), (14), (15), or (16). They correspond to our proposed model, the Goel-Okumoto model, the No-k model, the Only-G1 model, and the Yamada delayed S-shaped, respectively.Differentiatingχ2 with respect to each unknown parameter and setting the partial derivatives to zero, we can obtain the estimated values [34, 35]. Specifically, for our proposed model, we have(18)∂χ2∂a=∂χ2∂b=∂χ2∂p1=∂χ2∂p2=∂χ2∂k=0.Using numerical methods to solve the set of simultaneous equations above yields the estimation for unknown parameters in our proposed model.


## 3.3. Metrics for Model Comparison

With estimated parameters, we can conduct comparative analysis among models based on the following metrics:(1) Goodness of Fit (R2): the value of R2 determines how well the curve fits the real data [19, 36]. It is defined as follows:(19)R2=1-∑j=1Kmtj-mj2∑j=1Kmj-∑l=1Kml/K2.It measures the percentage of the total variance about the mean accounted for the fitted curve. It ranges from 0 to 1. The larger R2 is, the better the model fits data. In this work, the model is considered to provide a good fit if R2 is above 0.95.(2) Mean of Square Error (MSE): the value of MSE measures the average of squares of the differences between the predicted values and the actual values [9]. It is defined as follows:(20)MSE=1K∑j=1Kmtj-mj2.Generally, the closer to zero the MSE value is, the better the model fits the data.(3) Theil Statistic (TS): the value of TS indicates the average percentage deviation with respect to the actual failure data during the selected time period [37]. It is defined as follows:(21)TS=∑j=1Kmtj-mj2∑j=1Kyj2.A smaller TS value means a smaller deviation and a better prediction capability. In this work, the model is considered to provide a high-level prediction if the TS value is below 10%.(4) Relative Error (RE): the value of RE measures the relative difference between the predicted value and the real observations [25, 38]. The definition can be stated as follows:(22)RE=mtq-mqmq.Specifically, assuming that we have observed mq failures by the end of test at time tq, the failure data up to time te (te≤tq) will be used to estimate parameters in the models. Substituting the estimated parameters in the mean value function yields the estimation of the number of failures m(tq) by tq. The estimation is then compared with the actual number mq. Such procedure is repeated for various values of te. Positive values of error indicate overestimation and negative values indicate underestimation.


## 3.4. Estimation Results and Model Comparison

The parameter estimation can help characterize properties of theoretical models, determine the number of errors that have already been detected/corrected in the testing process, and predict the number of errors that will be eventually encountered by users. Specifically, by minimizing the objective function in (17), we can obtain the estimated results for each model.(i) For Apache 2.0.35, the estimated results are presented in Table2, with a^=74.8102, b^=0.0432, p1^=0.5905, p2^=0.2838, and k^=0.3554 for our proposed model. Then, the ratio of G1 errors to all types of errors is approximately 59%, the ratio of G2 errors is around 28%, and the leftover are G3 errors. Note that the errors consist of not only independent errors (G1), although G2 and G3 account much less than G1. Through incorporating such interdependency among generations of errors, we are able to strengthen the understanding towards composition of software system. Besides, we find that the estimated value of k is obviously larger than 0. Hence, not considering the effect of testing effort in the analysis of the reliability growth process may lead to significantly biased results.Table 2
Optimal estimated values: Apache 2.0.35.
a ^ b ^ p 1 ^ p 2 ^ p 3 ^ k ^ R 2 MSE TS Goel-Okumoto model 84.6047 0.0564 — — — — 0.9873 5.7624 0.0417 Yamada delayed S-shaped model 74.2741 0.1539 — — — — 0.9830 7.7033 0.0482 No-k model 76.4154 0.1087 0.1593 0.7293 0.111 4 ∗ — 0.9947 2.3912 0.0269 Only-G1 model 77.1477 0.0438 — — — 0.2323 0.9925 3.4129 0.0321 Proposed model 74.8102 0.0432 0.5905 0.2838 0.125 8 ∗ 0.3554 0.9954 2.0979 0.0252 ∗Calculated by p3^=1-p1^-p2^.Illustrations for the theoretical values versus their observations are shown in Figure2. We find that the theoretical values are highly consistent with the real-world observations. We can further examine the failure intensity function λ(t). Figure 3 shows that the failure intensity function is a single-peaked curve; that is, the error detection rate first increases and then decreases with time. It in turn implies that our proposed model falls into the S-shaped category for the SRGMs [34].Figure 2
Predicted values and real-world observations for Apache 2.0.35. Solid line: predicted values; dot: observations.Figure 3
Failure intensity function with respect to time, with estimated values based on Apache 2.0.35.We observe that our proposed model well fits the real-world software failure data. This can be seen from the high value ofR2, that is, 0.9954, which is very close to 1. By comparing with the Goel-Okumoto model, the Yamada delayed S-shaped model, the No-k model, and the Only-G1 model, R2 values are the highest in our proposed model. This conclusion can also be drawn by comparing MSE values among the three models; our proposed model provides the lowest MSE values (2.0979). From this aspect, the model in our paper best fits the observed data about Apache 2.0.35.Furthermore, the prediction capability in our model is very high. The low TS value for Apache 2.0.35, 0.0252, is much lower than 10% and well confirms a high prediction capability for our model. Compared with the benchmarks, our model can provide significantly lower TS values and higher predication capacity to the failure of Apache system. Lastly, the relative error in prediction using this data set is computed and results are plotted in Figure4. We observe that the relative error approaches 0 as te approaches tq and the relative error curve is usually within ±5%.Figure 4
Predicted values and real-world observations for Apache 2.0.36. Solid line: predicted values; dot: observations.(ii) For Apache 2.0.36, the estimated results are presented in Table3, with a^=49.4122, b^=0.0045, p1^=0.4031, p2^=0.2855, and k^=0.8967 for our proposed model. Three generations of errors account for 40%, 29%, and 31%, respectively. Obviously, G2 and G3 significantly contribute to the error composition, which counts for 60% in total. We also find that the estimated value of k is much larger than 0 implying that the testing-effort effect is necessary.Table 3
Optimal estimated values: Apache 2.0.36.
a ^ b ^ p 1 ^ p 2 ^ p 3 ^ k ^ R 2 MSE TS Goel-Okumoto model 52.3158 0.0393 — — — — 0.9526 8.8417 0.0707 Yamada delayed S-shaped model 49.8964 0.0896 — — — — 0.9550 8.3927 0.0689 No-k model 50.4507 0.0669 0.0000 0.8065 0.193 5 ∗ — 0.9706 5.4946 0.0557 Only-G1 model 50.1765 0.0196 — — — 0.3207 0.9643 6.6635 0.0614 Proposed model 49.4122 0.0045 0.4031 0.2855 0.311 4 ∗ 0.8967 0.9806 3.6163 0.0452 ∗Calculated by p3^=1-p1^-p2^.Then, we can obtain theoretical values in our proposed model. Illustrations for the theoretical values versus their observations are presented in Figure5, which shows that the theoretical values are consistent with the real-world data. In Figure 6, we present the failure intensity function λ(t) based on the estimated values. Similar to Apache 2.3.35, the hump-shaped curve shows that this function possesses an increasing-then-decreasing trend.Figure 5
Failure intensity function with respect to time, with estimated values based on Apache 2.0.36.Figure 6
Predicted values and real-world observations for Apache 2.0.39. Solid line: predicted values; dot: observations.We find that our proposed model also well fits the real-world software failure data for Apache 2.0.36. This can be seen from the high value ofR2, 0.9806. It is also much higher than those in the Goel-Okumoto model, the Yamada delayed S-shaped model, the No-k model, and the Only-G1 model. By comparing MSE values among the models, our proposed model provides the lowest MSE value (3.6163). From this aspect, the proposed model best fits the observed data about Apache 2.0.36.Furthermore, the TS value for Apache 2.0.36 is as low as 0.0452, which is much lower than 10% and well confirms a high prediction capability of our model. Compared with the benchmarks, our model shows a lower value of TS and a higher level of predication capability. Lastly, the relative error in prediction for this data set is computed, as shown in Figure7. We observe that the relative error approaches 0 as te approaches tq and the curve is usually within ±5%.Figure 7
Failure intensity function with respect to time, with estimated values based on Apache 2.0.39.(iii) For Apache 2.0.39, the estimated results are presented in Table4, with a^=57.1499, b^=0.0300, p1^=0.5299, p2^=0.3820, and k^=0.1940 for our proposed model. The ratios of three generations of errors become about 53%, 38%, and 9%, respectively. Then, the ratio for G1 errors is approximately 59%, the ratio for G2 errors is around 28%, and the leftover are G3 errors. We also find that the estimated value of k is significantly greater than 0. Thus, ignoring the effect of testing effort in the analysis of the software reliability growth curve would lead to significant bias.Table 4
Optimal estimated values: Apache 2.0.39.
a ^ b ^ p 1 ^ p 2 ^ p 3 ^ k ^ R 2 MSE TS Goel-Okumoto model 58.3827 0.0367 — — — — 0.9871 2.5670 0.0316 Yamada delayed S-shaped model 56.8974 0.0805 — — — — 0.9879 2.4045 0.0306 No-k model 57.3279 0.0577 0.2414 0.6741 0.084 5 ∗ — 0.9969 0.6133 0.0154 Only-G1 model 57.3266 0.0204 — — — 0.2453 0.9957 0.8562 0.0182 Proposed model 57.1499 0.0300 0.5299 0.3820 0.088 1 ∗ 0.1940 0.9973 0.5314 0.0144 ∗Calculated by p3^=1-p1^-p2^.Comparison between the fitted curve and the observed failure data is illustrated in Figure8, which indicates that the theoretical values and the real-world observations are highly consistent. In Figure 9, we present the failure intensity function based on the estimated values. Obviously, it is also a single-peaked curve: the error detection rate first increases and then decreases with time.Figure 8
Relative error for Apache 2.0.35.Figure 9
Relative error for Apache 2.0.36.The high value ofR2, 0.9937, suggests that our proposed model well fits the real-world software failure data for this version. By comparing with the Goel-Okumoto model, the Yamada delayed S-shaped model, the No-k model, and the Only-G1 model, R2 values are the highest in our proposed model. This conclusion can also be drawn by comparing MSE values among the models; our proposed model provides the lowest MSE value (0.5314). Therefore, our proposed model best fits the observed data about Apache 2.0.39.Furthermore, the TS value for Apache 2.0.39 is 0.0144. It is much lower than 10% and well confirms a high prediction capability for our model. Compared with the benchmarks, TS values in our model are significantly lower, and prediction capacity of the failure occurrence is also much greater. Lastly, the relative error in prediction for this data set is computed and results are plotted in Figure10. We observe that the relative error approaches 0 as te approaches tq and the error curve is usually within ±5%.Figure 10
Relative error for Apache 2.0.39.In summary, all of three tests using different versions of Apache failure data sets suggest that our proposed model well fits the real data and presents better performance than the benchmarks in error occurrence prediction. Besides, both the testing effort and the error interdependency effects play an important role in the parameter estimation.


## 4. Software Release Policy

The theoretical modeling can also determine whether the software is ready to release to users or how much more effort should be spent in further testing. In this section, we formally examine the software developer’s best release time and deliver sensitivity analysis regarding how the test effort effect and error interdependency affect the optimal time and the cost.


### 4.1. Optimal Release Time

When the failure process is modeled by a nonhomogeneous Poisson process with a mean value function,mt, a popular cost structure [15, 25, 27, 39] is applied as(23)CT=c1mT+c2m∞-mT+c3T,where T is the release time of the software to be determined. In addition, c1 is the expected cost of removing a fault during the testing phase, c2 is the expected cost of removing a fault during the operation phase, and c3 is the expected cost per unit time of testing. Specifically, the expected cost of removing a fault during the operation phase is higher than that during the testing phase; that is, c2>c1. Overall, the above cost function is the summation of the testing cost, including the failure cost during testing and actual cost of testing, and the cost of failure in the field.Proposition 1.
If only the cost is considered, the optimal release time satisfiesm′T∗=c3/c2-c1.Proof of Proposition1.
Note that the objective function is as follows:(24)CT=c1mT+c2m∞-mT+c3T.The first-order condition yields (25)C′T∗=c1m′T∗-c2m′T∗+c3=0,which indicates m′T∗=c3/c2-c1.It is also reasonable to quit testing as soon as the number of remaining faults in the system is less than a prescribed portion of total faults. Hence, a simple criterion for software release is given [8, 25]:(26)R1t=mta,which is the type-I reliability definition.Alternatively, given that the testing or the operation has been conducted up to timet, the probability that a software failure will not occur in the time interval (t,t+δ), with δ≥0, can be given by the following conditional probability [3, 40]:(27)R2δ∣t=e-mt+δ-mt,which is the type-II reliability definition.In most situations, the project managers need to strike a balance between the cost and the reliability level. That is, on one hand, the reliability level should be at least as high as a desired level set beforehand; on the other hand, given the target level of software reliabilityα, the cost minimization problem over the release time T should be imposed. Considering this tradeoff yields(28)R1T≥αfor the type-I reliability definition or (29)R2δ∣T≥αfor the type-II reliability definition.As an illustrative example, Figure11 presents how the software reliability and the total cost change with time. In this example, we use the data set with Apache 2.0.36 and take δ=5, c1=20$, c2=150$, and c3=15$. Suppose that the reliability must be higher than 0.95; that is, α=0.95. Then, the reliability constraint can be satisfied only if T≥T1=45.99 for the type-I reliability definition (or T≥T2=66.95 for the type-II reliability definition). When only the total cost is considered, we find that the cost is minimized at Tc=54.37. Furthermore, when both criteria (reliability and cost) are taken into account, we find that T∗=54.37 for the type-I reliability definition (or T∗=66.95 for the type-II reliability definition).Figure 11
The reliability and the cost as a function of time for Apache 2.0.36. Solid line: cost; dash-dot line: type-I definition of reliability; dash line: type-II definition of reliability; dot line: target level of reliability.In addition, for Apache 2.0.35, the reliability level is above 0.95 only ifT≥T1=30.91 if the type-I reliability definition is taken (or T≥T2=55.15 for the type-II reliability definition). When only the cost is considered, it is minimized at Tc=38.13. Furthermore, when both criteria (reliability and cost) are taken into account, we find that T∗=38.13 for the type-I reliability definition (or T∗=55.15 for the type-II reliability definition).Similarly, for Apache 2.0.39, the reliability level is higher than 0.95 only ifT≥T1=62.21 if the type-I reliability definition is taken (or T≥T2=103.20 for the type-II reliability definition). When only the cost is considered, the minimization is attained at Tc=61.61. When both reliability and cost are considered, we find that T∗=62.21 for the type-I reliability definition (or T∗=103.20 for the type-II reliability definition).


### 4.2. Sensitivity Analysis

As mentioned above, we have incorporated both the testing-effort and error interdependency effects into our proposed model. In this part, we intend to examine what if the testing effort, the error interdependency, or both of them are ignored in decision-making of software release. The first case corresponds to the No-k model (14), the second case corresponds to the Only-G1 model (15), and the third case is just the Goel-Okumoto model (13).For each model, we shall examine the optimal release time and the corresponding minimal cost when a desired reliability level is set beforehand. The data set with Apache 2.0.36 is used to illustrate our findings. Under the type-I reliability definition for reliability measurement, the optimal release time and the minimal cost as a function ofα are presented in Figures 12 and 13, respectively. Similarly, under the type-II reliability definition, the optimal release time and the minimal cost as a function of α are shown in Figures 14 and 15, respectively. Other parameters take the same values as analysis in Section 4.1; that is, δ=5, c1=20$, c2=150$, and c3=15$.Figure 12
Optimal release time as a function of desired reliability level, with estimated values based on Apache 2.0.36. The type-I reliability definition is adopted for such analysis. Solid line: our proposed mode; dot line: the Only-G1 model; dash-dot line: the No-k model; dash line: the Goel-Okumoto model.Figure 13
Total cost as a function of desired reliability level, with estimated values based on Apache 2.0.36. The type-I reliability definition is adopted for such analysis. Solid line: our proposed mode; dot line: the Only-G1 model; dash-dot line: the No-k model; dash line: the Goel-Okumoto model.Figure 14
Optimal release time as a function of desired reliability level, with estimated values based on Apache 2.0.36. The type-II reliability definition is adopted for such analysis. Solid line: our proposed mode; dot line: the Only-G1 model; dash-dot line: the No-k model; dash line: the Goel-Okumoto model.Figure 15
Total cost as a function of desired reliability level, with estimated values based on Apache 2.0.36. The type-II reliability definition is adopted for such analysis. Solid line: our proposed mode; dot line: the Only-G1 model; dash-dot line: the No-k model; dash line: the Goel-Okumoto model.Under either type-I or type-II reliability definition, both the optimal release time and the total cost would be mistakenly estimated if either testing effort or fault interdependency is ignored. Unlike these two cases, our proposed model always suggests an earlier release time and a lower testing cost.As an example, we compare the Goel-Okumoto model and our proposed model when the type-I reliability definition is adopted andα is set as 95%. The Goel-Okumoto model indicates the best release time to be 76.56 while that for our proposed model is 54.3. Hence, the testing lasts much longer if the Goel-Okumoto model is used to decide the stopping time. Similarly, using the Goel-Okumoto model yields the minimal cost as 2529$ while that for our proposed model is just 1896$, implying that a decision based on our model can save the software developers as much as 25% in cost.Alternatively, when the type-II reliability definition is adopted, the Goel-Okumoto model predicts the best release time as 132.2 while that for our proposed model is 66.75, which suggests that the testing duration in our model is doubled compared with traditional Goel-Okumoto model. Similarly, according to the Goel-Okumoto model, the minimal cost is 3083$ while that for our proposed model is merely 2000$. Thus, a decision based on our model can save the software developers as much as 35% in terms of the cost.


## 4.1. Optimal Release Time

When the failure process is modeled by a nonhomogeneous Poisson process with a mean value function,mt, a popular cost structure [15, 25, 27, 39] is applied as(23)CT=c1mT+c2m∞-mT+c3T,where T is the release time of the software to be determined. In addition, c1 is the expected cost of removing a fault during the testing phase, c2 is the expected cost of removing a fault during the operation phase, and c3 is the expected cost per unit time of testing. Specifically, the expected cost of removing a fault during the operation phase is higher than that during the testing phase; that is, c2>c1. Overall, the above cost function is the summation of the testing cost, including the failure cost during testing and actual cost of testing, and the cost of failure in the field.Proposition 1.
If only the cost is considered, the optimal release time satisfiesm′T∗=c3/c2-c1.Proof of Proposition1.
Note that the objective function is as follows:(24)CT=c1mT+c2m∞-mT+c3T.The first-order condition yields (25)C′T∗=c1m′T∗-c2m′T∗+c3=0,which indicates m′T∗=c3/c2-c1.It is also reasonable to quit testing as soon as the number of remaining faults in the system is less than a prescribed portion of total faults. Hence, a simple criterion for software release is given [8, 25]:(26)R1t=mta,which is the type-I reliability definition.Alternatively, given that the testing or the operation has been conducted up to timet, the probability that a software failure will not occur in the time interval (t,t+δ), with δ≥0, can be given by the following conditional probability [3, 40]:(27)R2δ∣t=e-mt+δ-mt,which is the type-II reliability definition.In most situations, the project managers need to strike a balance between the cost and the reliability level. That is, on one hand, the reliability level should be at least as high as a desired level set beforehand; on the other hand, given the target level of software reliabilityα, the cost minimization problem over the release time T should be imposed. Considering this tradeoff yields(28)R1T≥αfor the type-I reliability definition or (29)R2δ∣T≥αfor the type-II reliability definition.As an illustrative example, Figure11 presents how the software reliability and the total cost change with time. In this example, we use the data set with Apache 2.0.36 and take δ=5, c1=20$, c2=150$, and c3=15$. Suppose that the reliability must be higher than 0.95; that is, α=0.95. Then, the reliability constraint can be satisfied only if T≥T1=45.99 for the type-I reliability definition (or T≥T2=66.95 for the type-II reliability definition). When only the total cost is considered, we find that the cost is minimized at Tc=54.37. Furthermore, when both criteria (reliability and cost) are taken into account, we find that T∗=54.37 for the type-I reliability definition (or T∗=66.95 for the type-II reliability definition).Figure 11
The reliability and the cost as a function of time for Apache 2.0.36. Solid line: cost; dash-dot line: type-I definition of reliability; dash line: type-II definition of reliability; dot line: target level of reliability.In addition, for Apache 2.0.35, the reliability level is above 0.95 only ifT≥T1=30.91 if the type-I reliability definition is taken (or T≥T2=55.15 for the type-II reliability definition). When only the cost is considered, it is minimized at Tc=38.13. Furthermore, when both criteria (reliability and cost) are taken into account, we find that T∗=38.13 for the type-I reliability definition (or T∗=55.15 for the type-II reliability definition).Similarly, for Apache 2.0.39, the reliability level is higher than 0.95 only ifT≥T1=62.21 if the type-I reliability definition is taken (or T≥T2=103.20 for the type-II reliability definition). When only the cost is considered, the minimization is attained at Tc=61.61. When both reliability and cost are considered, we find that T∗=62.21 for the type-I reliability definition (or T∗=103.20 for the type-II reliability definition).


## 4.2. Sensitivity Analysis

As mentioned above, we have incorporated both the testing-effort and error interdependency effects into our proposed model. In this part, we intend to examine what if the testing effort, the error interdependency, or both of them are ignored in decision-making of software release. The first case corresponds to the No-k model (14), the second case corresponds to the Only-G1 model (15), and the third case is just the Goel-Okumoto model (13).For each model, we shall examine the optimal release time and the corresponding minimal cost when a desired reliability level is set beforehand. The data set with Apache 2.0.36 is used to illustrate our findings. Under the type-I reliability definition for reliability measurement, the optimal release time and the minimal cost as a function ofα are presented in Figures 12 and 13, respectively. Similarly, under the type-II reliability definition, the optimal release time and the minimal cost as a function of α are shown in Figures 14 and 15, respectively. Other parameters take the same values as analysis in Section 4.1; that is, δ=5, c1=20$, c2=150$, and c3=15$.Figure 12
Optimal release time as a function of desired reliability level, with estimated values based on Apache 2.0.36. The type-I reliability definition is adopted for such analysis. Solid line: our proposed mode; dot line: the Only-G1 model; dash-dot line: the No-k model; dash line: the Goel-Okumoto model.Figure 13
Total cost as a function of desired reliability level, with estimated values based on Apache 2.0.36. The type-I reliability definition is adopted for such analysis. Solid line: our proposed mode; dot line: the Only-G1 model; dash-dot line: the No-k model; dash line: the Goel-Okumoto model.Figure 14
Optimal release time as a function of desired reliability level, with estimated values based on Apache 2.0.36. The type-II reliability definition is adopted for such analysis. Solid line: our proposed mode; dot line: the Only-G1 model; dash-dot line: the No-k model; dash line: the Goel-Okumoto model.Figure 15
Total cost as a function of desired reliability level, with estimated values based on Apache 2.0.36. The type-II reliability definition is adopted for such analysis. Solid line: our proposed mode; dot line: the Only-G1 model; dash-dot line: the No-k model; dash line: the Goel-Okumoto model.Under either type-I or type-II reliability definition, both the optimal release time and the total cost would be mistakenly estimated if either testing effort or fault interdependency is ignored. Unlike these two cases, our proposed model always suggests an earlier release time and a lower testing cost.As an example, we compare the Goel-Okumoto model and our proposed model when the type-I reliability definition is adopted andα is set as 95%. The Goel-Okumoto model indicates the best release time to be 76.56 while that for our proposed model is 54.3. Hence, the testing lasts much longer if the Goel-Okumoto model is used to decide the stopping time. Similarly, using the Goel-Okumoto model yields the minimal cost as 2529$ while that for our proposed model is just 1896$, implying that a decision based on our model can save the software developers as much as 25% in cost.Alternatively, when the type-II reliability definition is adopted, the Goel-Okumoto model predicts the best release time as 132.2 while that for our proposed model is 66.75, which suggests that the testing duration in our model is doubled compared with traditional Goel-Okumoto model. Similarly, according to the Goel-Okumoto model, the minimal cost is 3083$ while that for our proposed model is merely 2000$. Thus, a decision based on our model can save the software developers as much as 35% in terms of the cost.


## 5. Concluding Remarks

In the development process, a variety of tests need to be conducted to ensure the reliability of the software system. In this work, we propose a framework for software reliability modeling that simultaneously captures effects of testing effort and interdependence between error generations. In order to evaluate our proposed model, we compare its performance with other existing models by doing experiments with actual software failure data, that is, three versions of Apache. The nonlinear least square estimation technique is applied for parameter estimation, and values ofR2, mean of square error, Theil Statistic, and relative error are used for model comparison in terms of fitting quality and prediction capability. The numerical analysis from all three Apache versions demonstrates that the proposed model fits the real-world data very well and exhibits a high level of prediction capability.Our theoretical results can help software project managers assess the most cost-effective time to stop software testing and to release the product. Specifically, we have formally examined the release timing and provided useful managerial insights based on the Apache data sets. Moreover, our proposed model recommends an earlier release time and a reduced cost compared with the Goel-Okumoto model and the Yamada delayed S-shaped model. Therefore, our modeling essentially enhances accuracy of the understanding towards the software reliability growth process and the best time to release a software version.



---

*Source: 9276093-2016-11-13.xml*