# Two-Dimensional Software Defect Models with Test Execution History

Authors: Tomotaka Ishii; Tadashi Dohi
Journal: ISRN Software Engineering
 (2012)

---

## Abstract

In general, the software-testing time may be measured by two kinds of time scales: calendar time and test execution time. In this paper, we develop two-dimensional software reliability models with two-time measures and incorporate both of them to assess the software reliability with higher accuracy. Since the resulting software defect models are based on the familiar nonhomogeneous Poisson processes with two time scales, which are the natural extensions of one-dimensional software defect models, it is possible to treat the time data both simultaneously and effectively. We investigate the dependence of test-execution time as a testing effort on the software reliability assessment and validate quantitatively the software defect models with two-time scales. We also consider an optimization problem when to stop the software testing in terms of two-time measurements.


---

## Body

## 1. Introduction

The reliable software plays a central role to develop the dependable and high assurance computer-based systems. Since the debugging cycle times of software are often reduced due to smaller release time requirements, the accurate estimation of software reliability tends to be more important day by day, especially, in the earlier testing phase. Software defect models (SDMs) are used to measure the quantitative software reliability and to control the software testing process effectively. Since the software reliability is defined as the probability that software failures caused by faults do not occur for a specified time period, the time evolution of failure occurrence process (fault detection process in software testing) should be modeled by any stochastic counting process. In fact, during the last three decades since the seminal contribution by Jelinski and Moranda [1], a huge number of SDMs have been extensively developed by many authors to help us in estimating the number of initial fault contents and understanding the effect of errors on software operation as well as in predicting the software reliability [2–5]. These are characterized by modeling the software intensity function which implies the instantaneous debugging rate of software faults in the testing phase and is equivalent to the transition rate of stochastic point process.When one counts the number of software faults detected in the testing phase, the cumulative number of faults is described by a generalized pure birth process with time and/or state dependent transition rate [6, 7]. If the software intensity function depends on only the number of remaining faults, it is reduced to a time-homogeneous continuous-time Markov chain (CTMC). The Jelinski and Moranda SRM [1] is the simplest example of time-homogeneous CTMCs. On the other hand, Goel and Okumoto [8] develop an alternative CTMC model with time-dependent transition rate, based on the nonhomogeneous Poisson process (NHPP). Since the NHPP has a simple mathematical structure and its analytical treatment is quite easy, many SDMs belonging to this category are developed under different software debugging scenarios with deterministic intensity functions. Langberg and Singpurwalla [9] provided a bridge between Jelinski and Moranda SRM [1] and Goel and Okumoto SRM [8] from the Bayesian point of view. Miller [10] extended the Langberg and Singpurwalla’s idea to more general cases with the concept of exponential order statistics. Gokhale and Trivedi [11] took account of the testing coverage and propose a different unification framework of the NHPP-based SRMs. Apart from the sophisticated probabilistic approaches, Huang et al. [12] explained the deterministic behavior of NHPP-based SDMs, namely, the mean value functions, by introducing several kinds of mean operations. Pham and Zhang [13] solved a generalized differential equation by which the mean value function is governed and proposed a unified NHPP-based SDM with many parameters.The above modeling frameworks may unify the NHPP-based SDMs, but never refer to how to use them effectively in practice. For instance, the software testing time may be usually measured by two kinds of time scales: calendar time (hour, day, week, etc.) and test-execution time (CPU second, CPU minute, CPU hour, etc). Musa et al. [3] developed the calendar-time theory and gave a relationship between the calendar-time modeling and the test-execution time modeling in software reliability. Parr [14] used the Rayleigh curve to describe the software development effort (see [15, 16] for the other static approaches). An alternative approach based on the NHPP-based SDMs was proposed by Yamada et al. [17, 18], Huang and Kuo [19], Huang [20], Kuo et al. [21]. They defined the mean value functions of NHPPs as functions of test-execution time and developed some testing-efforts dependent SDMs. These SDMs are quite simple for practical use, but involve several technical problems in terms of parameter estimation and model validation, as we point out them latter. Chen et al. [22] took both execution time, and code coverage into consideration as factors that influence the cumulative number of software failure occurrences. Grottke [23] considered a number of factors influencing the cumulative number of failure occurrences, like calendar time, execution time and code coverage, by specifying consecutive relationships, within the framework of CTMC modeling.In this paper, we summarize two-dimensional SDMs with two-time measures and incorporate both of them to assess the quantitative software reliability with higher accuracy. Ishii and Dohi [24] developed two-dimensional NHPP-based SDMs and estimate the model parameters with elementary least squares estimation. In the subsequent paper [25], the same authors considered the maximum likelihood estimation for the same SDMs and investigated the goodness-of-fit and the predictive performance with real software fault data. Ishii et al. [26] focused on the discrete-time models with the number of test cases spent for testing and proposed the similar two-dimensional SDMs. The modeling framework developed in [24–26] was widely used for obtaining analogous SDMs [27, 28]. Since the resulting SDMs are based on the familiar NHPPs with two time scales which are the natural extensions of one-dimensional SDMs, it is possible to treat the time data both simultaneously and effectively. In that sense, other two-dimensional SDMs are quite different from Grottke [23]. Also it is worth noting that the two-dimensional SDMs in [25, 26] should be distinguished from the earlier stochastic models developed in the reliability filed [29–33], because they are consistent to the existing one-dimensional NHPP-based SDMs in the literature and involve as special cases. The modeling approach employed here enables us to apply the maximum likelihood method for estimating the model parameters as well as to provide a useful tool to incorporate the testing-effort expenditure as one of the testing matrices. As pointed out by Musa et al. [3], the test-execution time is the best time scale to measure the net testing time and efforts. On the other hand, the software reliability should be defined on the calendar time based on the operational profile, and the testing control to release the software products will be also made on the calendar time. That is, it is essentially important to treat two kinds of time measures in the software reliability assessment. We investigate the dependence of test-execution time as a testing effort on the software reliability assessment and validate quantitatively our new SDMs with two-time scales. We also consider an optimization problem when to stop the software testing in terms of two-time measurements.


## 2. One-Dimensional Software Defect Modeling

### 2.1. NHPP-Based SDMs

We make the following assumptions.(A-1)
The software fault detected in testing phase is instantly fixed and removed.(A-2)
The number of initial fault contents in the program,N(≥0), is given by a nonnegative discrete random variable having the probability mass function (p.m.f.) g(n)=Pr{N=n}.(A-3)
Thejth software fault (j=1,2,3,…) can be detected by independent and identically distributed (i.i.d.) random calendar time Tj, having the cumulative distribution function (c.d.f.) F(t)=Pr{Tj≤t}.We call the above c.d.f.the fault-detection probability in this paper. From the assumptions (A-1)–(A-3), it can be seen that the total number of software faults detected by an arbitrary calendar time t (day, week, month, etc.) is given by the univariate random variable X(t) having the conditional binomial p.m.f.
(1)Pr{X(t)=x∣N=n}=(nx)F(t)x{1-F(t)}n-x,
where
(2)E[X(t)∣N=n]=nF(t),Var[X(t)∣N=n]=nF(t){1-F(t)}.
If F(t) is the exponential distribution and T(1),T(2),T(3),… are the exponential order statistics sampled from F(t), then the corresponding stochastic point process {X(t),t≥0}=max{j≥0;t≥T(j)} is equivalent to the well-known Jelinski and Moranda SDM [1].In addition, suppose that the initial number of software faultsN is unknown and obeys the Poisson distribution with mean ω (>0), that is,
(3)g(n)=ωnn!exp{-ω}.
Then the cumulative number of software faults detected by time t is given by
(4)Pr{X(t)=x}=∑n=0∞‍Pr{X(t)=x∣N=n}g(n)={ωF(t)}xx!exp{-ωF(t)}.
This implies that the probabilistic law of X(t) is governed by the well-known nonhomogeneous Poisson process (NHPP) with mean value function Λ(t)=ωF(t), and that E[X(t)]=Var[X(t)]=Λ(t). By substituting several kinds of fault-detection probabilities into F(t), it is possible to develop a number of NHPP-based SDMs. Goel and Okumoto [8] assumed the exponential distribution F(t)=1-exp(-at) with constant a(>0) and developed the exponential NHPP-based SDM with Λ(t)=ω{1-exp(-at)}. Yamada et al. [34] and Goel [35] used the two-phase Erlang distribution F(t)=1-(1+at)exp(-at) and the Weibull distribution F(t)=1-exp(-atb)(b>0) and proposed the S-shaped NHPP-based SDM and the generalized exponential NHPP-based SDM with Λ(t)=ω{1-(1+at)exp(-at)} and Λ(t)=ω{1-exp(-atb)}, respectively. In Figure 1, we show the configuration of one-dimensional software debugging with the i.i.d. fault-detection times to obtain the NHPP-based SDMs.Figure 1
Configuration of one-dimensional software debugging.On the other hand, when the cumulative test-execution times (CPU time) is observed in the testing phase, it can be used as an alternative of the calendar time, that is, Λ(s)=ωF(s). Musa et al. [3] claimed to use the test-execution time effectively instead of the calendar time in software reliability modeling and pointed out that the execution-time modeling is superior to the calendar-time modeling. However, the execution-time modeling has a limitation to the real software reliability assessment and management. To overcome the problem, Musa et al. [3] developed the calendar-time modeling in order to transform the execution time to the calendar time and gave a relationship (deterministic formula) among test-execution time, calendar time, and resource usage parameters through a simple linear regression analysis. Unfortunately, since this is an intuitive model under a specific testing circumstance, it is not applicable to the general reliability assessment in practice. In other words, the calendar-time modeling by Musa et al. [3] involves some theoretical problems and has seldom been used in the actual software reliability management.


### 2.2. Testing-Effort Dependent SDMs

It is common to assume that the cumulative test-execution time is regarded as a function of calendar time, that is,s=W(t). Yamada et al. [17, 18] assumed the fault-detection probability F(t)=1-exp{-rW(t)} and proposed the testing-effort dependent NHPP-based SDMs, where r (>0) is a constant. More specifically, define the function
(5)W(t)=∫0t‍w(x)dx,W(0)=0,
where w(t) is called the testing-effort function and means the testing-effort expenditure spent per unit calendar time. From (5), the function W(t) can be considered as the cumulative testing-effort expenditure consumed up to time t. Yamada et al. [17, 18] further assumed the Weibull testing-effort function
(6)wW(t)=αβmtm-1e-βtm
with arbitrary parameters α (>0), β (>0), and m (>0). Huang and Kuo [19] and Kuo et al. [21] proposed the logistic testing-effort function
(7)wL(t)=α1+βe-mt.Lettj (=j), sj, and xj denote the calendar time, the test-execution time, and the cumulative number of software faults detected by the jth calendar week, respectively. Given n fault data (tj,sj,xj)(j=1,2,…,n), the parameters (α,β,m) or (α,m) in (6) and (7) are estimated by means of the regression analysis with the data (tj,sj), and the remaining parameters (ω,r) are done by the method of maximum likelihood with the data (tj,xj). Since this two-stage method is a combination of the least squares method and the maximum likelihood method, the resulting estimates have no significant meaning in terms of statistical theory. That is, the maximum likelihood estimates of (ω,r) with fixed (α,β,m) do not possess the invariant property and/or asymptotical efficiency. In addition, since the testing resource is finite and limt→∞W(t)<1, it is evident that limt→∞F(t)=1-exp(-r)<1 and that the fault detection probability is defective [36]. This implies that the initial fault contents are given by the Poisson distributed random variable with mean ω{1-exp(-r)} and is inconsistent with the model assumptions in II-A. In the testing-effort dependent modeling, the fundamental idea is that the test-execution time s=W(t) is given by a function of the calendar time t. However, if the execution time s can be observed directly, substituting s into W(t) formally leads to Λ(s)=ω{1-exp(-rs)} which is equivalent to the mean value function of the simple exponential NHPP-based SDM with test-execution time s. Consequently, if one uses only the execution time s, it corresponds to the execution-time modeling based on the exponential NHPP and is independent of the calendar-time data. In this way, the classical testing-effort dependent SDMs [17–19, 21] give no satisfactory modeling framework to deal with two-time measures consistently in the past literature.


## 2.1. NHPP-Based SDMs

We make the following assumptions.(A-1)
The software fault detected in testing phase is instantly fixed and removed.(A-2)
The number of initial fault contents in the program,N(≥0), is given by a nonnegative discrete random variable having the probability mass function (p.m.f.) g(n)=Pr{N=n}.(A-3)
Thejth software fault (j=1,2,3,…) can be detected by independent and identically distributed (i.i.d.) random calendar time Tj, having the cumulative distribution function (c.d.f.) F(t)=Pr{Tj≤t}.We call the above c.d.f.the fault-detection probability in this paper. From the assumptions (A-1)–(A-3), it can be seen that the total number of software faults detected by an arbitrary calendar time t (day, week, month, etc.) is given by the univariate random variable X(t) having the conditional binomial p.m.f.
(1)Pr{X(t)=x∣N=n}=(nx)F(t)x{1-F(t)}n-x,
where
(2)E[X(t)∣N=n]=nF(t),Var[X(t)∣N=n]=nF(t){1-F(t)}.
If F(t) is the exponential distribution and T(1),T(2),T(3),… are the exponential order statistics sampled from F(t), then the corresponding stochastic point process {X(t),t≥0}=max{j≥0;t≥T(j)} is equivalent to the well-known Jelinski and Moranda SDM [1].In addition, suppose that the initial number of software faultsN is unknown and obeys the Poisson distribution with mean ω (>0), that is,
(3)g(n)=ωnn!exp{-ω}.
Then the cumulative number of software faults detected by time t is given by
(4)Pr{X(t)=x}=∑n=0∞‍Pr{X(t)=x∣N=n}g(n)={ωF(t)}xx!exp{-ωF(t)}.
This implies that the probabilistic law of X(t) is governed by the well-known nonhomogeneous Poisson process (NHPP) with mean value function Λ(t)=ωF(t), and that E[X(t)]=Var[X(t)]=Λ(t). By substituting several kinds of fault-detection probabilities into F(t), it is possible to develop a number of NHPP-based SDMs. Goel and Okumoto [8] assumed the exponential distribution F(t)=1-exp(-at) with constant a(>0) and developed the exponential NHPP-based SDM with Λ(t)=ω{1-exp(-at)}. Yamada et al. [34] and Goel [35] used the two-phase Erlang distribution F(t)=1-(1+at)exp(-at) and the Weibull distribution F(t)=1-exp(-atb)(b>0) and proposed the S-shaped NHPP-based SDM and the generalized exponential NHPP-based SDM with Λ(t)=ω{1-(1+at)exp(-at)} and Λ(t)=ω{1-exp(-atb)}, respectively. In Figure 1, we show the configuration of one-dimensional software debugging with the i.i.d. fault-detection times to obtain the NHPP-based SDMs.Figure 1
Configuration of one-dimensional software debugging.On the other hand, when the cumulative test-execution times (CPU time) is observed in the testing phase, it can be used as an alternative of the calendar time, that is, Λ(s)=ωF(s). Musa et al. [3] claimed to use the test-execution time effectively instead of the calendar time in software reliability modeling and pointed out that the execution-time modeling is superior to the calendar-time modeling. However, the execution-time modeling has a limitation to the real software reliability assessment and management. To overcome the problem, Musa et al. [3] developed the calendar-time modeling in order to transform the execution time to the calendar time and gave a relationship (deterministic formula) among test-execution time, calendar time, and resource usage parameters through a simple linear regression analysis. Unfortunately, since this is an intuitive model under a specific testing circumstance, it is not applicable to the general reliability assessment in practice. In other words, the calendar-time modeling by Musa et al. [3] involves some theoretical problems and has seldom been used in the actual software reliability management.


## 2.2. Testing-Effort Dependent SDMs

It is common to assume that the cumulative test-execution time is regarded as a function of calendar time, that is,s=W(t). Yamada et al. [17, 18] assumed the fault-detection probability F(t)=1-exp{-rW(t)} and proposed the testing-effort dependent NHPP-based SDMs, where r (>0) is a constant. More specifically, define the function
(5)W(t)=∫0t‍w(x)dx,W(0)=0,
where w(t) is called the testing-effort function and means the testing-effort expenditure spent per unit calendar time. From (5), the function W(t) can be considered as the cumulative testing-effort expenditure consumed up to time t. Yamada et al. [17, 18] further assumed the Weibull testing-effort function
(6)wW(t)=αβmtm-1e-βtm
with arbitrary parameters α (>0), β (>0), and m (>0). Huang and Kuo [19] and Kuo et al. [21] proposed the logistic testing-effort function
(7)wL(t)=α1+βe-mt.Lettj (=j), sj, and xj denote the calendar time, the test-execution time, and the cumulative number of software faults detected by the jth calendar week, respectively. Given n fault data (tj,sj,xj)(j=1,2,…,n), the parameters (α,β,m) or (α,m) in (6) and (7) are estimated by means of the regression analysis with the data (tj,sj), and the remaining parameters (ω,r) are done by the method of maximum likelihood with the data (tj,xj). Since this two-stage method is a combination of the least squares method and the maximum likelihood method, the resulting estimates have no significant meaning in terms of statistical theory. That is, the maximum likelihood estimates of (ω,r) with fixed (α,β,m) do not possess the invariant property and/or asymptotical efficiency. In addition, since the testing resource is finite and limt→∞W(t)<1, it is evident that limt→∞F(t)=1-exp(-r)<1 and that the fault detection probability is defective [36]. This implies that the initial fault contents are given by the Poisson distributed random variable with mean ω{1-exp(-r)} and is inconsistent with the model assumptions in II-A. In the testing-effort dependent modeling, the fundamental idea is that the test-execution time s=W(t) is given by a function of the calendar time t. However, if the execution time s can be observed directly, substituting s into W(t) formally leads to Λ(s)=ω{1-exp(-rs)} which is equivalent to the mean value function of the simple exponential NHPP-based SDM with test-execution time s. Consequently, if one uses only the execution time s, it corresponds to the execution-time modeling based on the exponential NHPP and is independent of the calendar-time data. In this way, the classical testing-effort dependent SDMs [17–19, 21] give no satisfactory modeling framework to deal with two-time measures consistently in the past literature.


## 3. Two-Dimensional SDMs

### 3.1. Bivariate Fault-Detection Time Distributions

To cope with two-time scales (t,s) simultaneously, we extend the above univariate binomial modeling framework to the bivariate one. Let T and S be the calendar time (week) and the test-execution time (CPU hr), respectively. Suppose that (T,S) is a continuous bivariate random variable having the bivariate c.d.f. [37, 38]
(8)Pr{T≤t,S≤s}=F(t,s),
where F(∞,∞)=1 and F(0,s)=F(t,0)=F(0,0)=0. For convenience, letting
(9)F(t,∞)=FT(t),F(∞,s)=FS(s)
denote the marginal distributions, it is worth noting in the bivariate probability distribution [37] that
(10)Pr{T>t,S>s}=1-FT(t)-FS(s)+F(t,s).
Also, it can be easily checked that Pr{T>t,S>s}≠1-F(t,s) and that the bivariate c.d.f. cannot be determined uniquely only by its marginal distributions, that is, the bivariate c.d.f. with the same marginal distributions can be defined infinitely (see [39–43]).In this paper, we focus on two simplest cases where(T,S) obeys the bivariate exponential distribution by Marshall and Olkin (Marshall-Olkin distribution) [42, 43] and the bivariate Weibull distribution by Lu and Bhattacharyya [44], because the objective here is not to develop a number of SRMs with different fault-detection probabilities.  (i)
Marshall-Olkin distribution:(11)FMO(t,s)=1-e-(a+c)t-e-(b+c)s+e-at-bs-c·max(t,s).(ii)
Bivariate Weibull distribution:(12)FLB(t,s)=1-e-(t/θ1)β1-e-(s/θ2)β2+e-{(t/θ1)β1/δ+(s/θ2)β2/δ}δ,where a (>0), b (>0), c∈[0,1], θ1 (>0), θ2 (>0), β1 (>0), β2 (>0), and δ∈(0,1] are arbitrary parameters. Figure 2 depicts the configuration of two-dimensional software debugging with two-time measures.Figure 2
Configuration of two-dimensional software debugging.For the initial fault contentsN, let {X(t,s),t≥0,s≥0} be the i.i.d. bivariate random variable and the number of software faults detected by time (t,s). Since each software fault-detection time distribution is given by F(t,s), from the assumptions (A-1)–(A-3), it is immediate to see that
(13)Pr{X(t,s)=x∣N=n}=(nx)F(t,s)x×{1-F(t,s)}n-x,Pr{X(t,s)=x}={ωF(t,s)}xx!exp{-ωF(t,s)}.
This is a two-dimensional NHPP with two-time scales t and s. In this paper, we call this type of NHPP the two-dimensional SDM with mean value function Λ(t,s)=E[X(t,s)]=Var[X(t,s)]=ωF(t,s).It is reasonable to assume that the test-execution time is proportional to the number of test cases. In the two-dimensional SDM, if eitherT or S takes an extremely large value, that is, a number of test cases are consumed during an extremely short period or very a few test cases are tried for an extremely long period, then the fault-detection probability becomes small. This satisfies our intuition for the relationship between the testing time and the number of test cases. It can be found that c=0 and F(t,s)=FT(t)FS(s) if T and S are statistically independent from each other. This means that the cumulative number of test cases has no correlation with the length of testing period measured by calendar time and seems to be quite unreasonable. In fact, we expect in modeling that the functions E[S∣T=t] and E[T∣S=s] are both increasing functions of t and s, respectively. This is the main reason to use the Marshall-Olkin distribution [42, 43], because the other bivariate exponential distributions with explicit form of c.d.f. such as the Gumbel distribution [40] have the negative correlation [27, 28]. In the Marshall-Olkin distribution, we have
(14)E[S∣T=t]=∫0∞‍Pr{S>s∣T=t}ds=1b-c(a+b+c)e-btb(a+c)(b+c),(15)E[T∣S=s]=∫0∞‍Pr{T>t∣S=s}ds=1a-c(a+b+c)e-asa(a+c)(b+c).
Also, it holds in this case that Pr{T=S}=c/(a+b+c)>0 and that the software fault can be detected with positive probability at time t=T=s.On the other hand, for the bivariate Weibull distribution in (12), we have
(16)Pr{S>s∣T=t}=∂FBW(t,s)/∂t∂2FBW(t,s)/∂t∂s=1θ1{(tθ1)β1/δe(t/θ1)1β-[(t/θ1)β1/δ+(s/θ2)β2/δ]δ×β1(tθ1)-1+(β1/δ)×[(tθ1)β1/δ+(sθ2)β2/δ]-1+δ}.
Unfortunately, the explicit forms of E[S∣T=t] and E[T∣S=s] for the bivariate Weibull distribution are not available, but can be numerically calculated. These regression functions may be used to predict the cumulative test-execution time in future at an arbitrary calendar time, after estimating model parameters in Λ(t,s).


### 3.2. Maximum Likelihood Estimation

In the seminal paper [24], we apply the least squares estimation to determine the model parameters. However, the advantage of the two-dimensional SDMs is to make it easy to estimate the model parameters by means of the maximum likelihood method. Let xj(j=1,2,…,n) denote the cumulative number of software faults detected by the jth calendar week. Given n fault data (tj,sj,xj)(j=1,2,…,n), the problem is to estimate the model parameter θ, where Λ(t,s;θ)=ωF(t,s;π), θ=(ω,π), π=(a,b,c), and (θ1,θ2,β1,β2,δ) for the Marshall-Olkin distribution and the bivariate Weibull distribution, respectively. Then, the likelihood function is given by
(17)L(θ)=Pr{X(t1,s1)=x1,X(t2,s2)=x2,…,X(tk,sk)=xk}=∏j=1k‍exp{-[Λ(tj,sj;θ)-Λ(tj-1,sj-1;θ)]}×{Λ(tj,sj;θ)-Λ(tj-1,sj-1;θ)}(xj-xj-1)(xj-xj-1)!=exp{-Λ(tk,sk;θ)}×∏j=1k‍{Λ(tj,sj;θ)-Λ(tj-1,sj-1;θ)}(xj-xj-1)(xj-xj-1)!.
Taking the logarithm of both sides of (17) yields the following logarithmic likelihood function:
(18)lnL(θ)=∑j=1k‍(xj-xj-1)×ln{Λ(tj,sj;θ)-Λ(tj-1,sj-1;θ)}-Λ(tk,sk;θ)-∑j=1k‍ln{(xj-xj-1)!},
where (t0,s0,x0)=(0,0,0).Since the logarithmic function is monotonically increasing, the maximum likelihood estimate of the parameterθmaximizes the logarithmic likelihood function lnL(θ). If the function lnL(θ) is strictly concave in θ, then the maximum likelihood estimate has to satisfy the first order condition of optimality ∂lnL(θ)/∂θ=0, where 0 is the zero vector. The above modeling approach provides a valid parameter estimation procedure based on the maximum likelihood method, without the two-stage procedure based on the least square method.


## 3.1. Bivariate Fault-Detection Time Distributions

To cope with two-time scales (t,s) simultaneously, we extend the above univariate binomial modeling framework to the bivariate one. Let T and S be the calendar time (week) and the test-execution time (CPU hr), respectively. Suppose that (T,S) is a continuous bivariate random variable having the bivariate c.d.f. [37, 38]
(8)Pr{T≤t,S≤s}=F(t,s),
where F(∞,∞)=1 and F(0,s)=F(t,0)=F(0,0)=0. For convenience, letting
(9)F(t,∞)=FT(t),F(∞,s)=FS(s)
denote the marginal distributions, it is worth noting in the bivariate probability distribution [37] that
(10)Pr{T>t,S>s}=1-FT(t)-FS(s)+F(t,s).
Also, it can be easily checked that Pr{T>t,S>s}≠1-F(t,s) and that the bivariate c.d.f. cannot be determined uniquely only by its marginal distributions, that is, the bivariate c.d.f. with the same marginal distributions can be defined infinitely (see [39–43]).In this paper, we focus on two simplest cases where(T,S) obeys the bivariate exponential distribution by Marshall and Olkin (Marshall-Olkin distribution) [42, 43] and the bivariate Weibull distribution by Lu and Bhattacharyya [44], because the objective here is not to develop a number of SRMs with different fault-detection probabilities.  (i)
Marshall-Olkin distribution:(11)FMO(t,s)=1-e-(a+c)t-e-(b+c)s+e-at-bs-c·max(t,s).(ii)
Bivariate Weibull distribution:(12)FLB(t,s)=1-e-(t/θ1)β1-e-(s/θ2)β2+e-{(t/θ1)β1/δ+(s/θ2)β2/δ}δ,where a (>0), b (>0), c∈[0,1], θ1 (>0), θ2 (>0), β1 (>0), β2 (>0), and δ∈(0,1] are arbitrary parameters. Figure 2 depicts the configuration of two-dimensional software debugging with two-time measures.Figure 2
Configuration of two-dimensional software debugging.For the initial fault contentsN, let {X(t,s),t≥0,s≥0} be the i.i.d. bivariate random variable and the number of software faults detected by time (t,s). Since each software fault-detection time distribution is given by F(t,s), from the assumptions (A-1)–(A-3), it is immediate to see that
(13)Pr{X(t,s)=x∣N=n}=(nx)F(t,s)x×{1-F(t,s)}n-x,Pr{X(t,s)=x}={ωF(t,s)}xx!exp{-ωF(t,s)}.
This is a two-dimensional NHPP with two-time scales t and s. In this paper, we call this type of NHPP the two-dimensional SDM with mean value function Λ(t,s)=E[X(t,s)]=Var[X(t,s)]=ωF(t,s).It is reasonable to assume that the test-execution time is proportional to the number of test cases. In the two-dimensional SDM, if eitherT or S takes an extremely large value, that is, a number of test cases are consumed during an extremely short period or very a few test cases are tried for an extremely long period, then the fault-detection probability becomes small. This satisfies our intuition for the relationship between the testing time and the number of test cases. It can be found that c=0 and F(t,s)=FT(t)FS(s) if T and S are statistically independent from each other. This means that the cumulative number of test cases has no correlation with the length of testing period measured by calendar time and seems to be quite unreasonable. In fact, we expect in modeling that the functions E[S∣T=t] and E[T∣S=s] are both increasing functions of t and s, respectively. This is the main reason to use the Marshall-Olkin distribution [42, 43], because the other bivariate exponential distributions with explicit form of c.d.f. such as the Gumbel distribution [40] have the negative correlation [27, 28]. In the Marshall-Olkin distribution, we have
(14)E[S∣T=t]=∫0∞‍Pr{S>s∣T=t}ds=1b-c(a+b+c)e-btb(a+c)(b+c),(15)E[T∣S=s]=∫0∞‍Pr{T>t∣S=s}ds=1a-c(a+b+c)e-asa(a+c)(b+c).
Also, it holds in this case that Pr{T=S}=c/(a+b+c)>0 and that the software fault can be detected with positive probability at time t=T=s.On the other hand, for the bivariate Weibull distribution in (12), we have
(16)Pr{S>s∣T=t}=∂FBW(t,s)/∂t∂2FBW(t,s)/∂t∂s=1θ1{(tθ1)β1/δe(t/θ1)1β-[(t/θ1)β1/δ+(s/θ2)β2/δ]δ×β1(tθ1)-1+(β1/δ)×[(tθ1)β1/δ+(sθ2)β2/δ]-1+δ}.
Unfortunately, the explicit forms of E[S∣T=t] and E[T∣S=s] for the bivariate Weibull distribution are not available, but can be numerically calculated. These regression functions may be used to predict the cumulative test-execution time in future at an arbitrary calendar time, after estimating model parameters in Λ(t,s).


## 3.2. Maximum Likelihood Estimation

In the seminal paper [24], we apply the least squares estimation to determine the model parameters. However, the advantage of the two-dimensional SDMs is to make it easy to estimate the model parameters by means of the maximum likelihood method. Let xj(j=1,2,…,n) denote the cumulative number of software faults detected by the jth calendar week. Given n fault data (tj,sj,xj)(j=1,2,…,n), the problem is to estimate the model parameter θ, where Λ(t,s;θ)=ωF(t,s;π), θ=(ω,π), π=(a,b,c), and (θ1,θ2,β1,β2,δ) for the Marshall-Olkin distribution and the bivariate Weibull distribution, respectively. Then, the likelihood function is given by
(17)L(θ)=Pr{X(t1,s1)=x1,X(t2,s2)=x2,…,X(tk,sk)=xk}=∏j=1k‍exp{-[Λ(tj,sj;θ)-Λ(tj-1,sj-1;θ)]}×{Λ(tj,sj;θ)-Λ(tj-1,sj-1;θ)}(xj-xj-1)(xj-xj-1)!=exp{-Λ(tk,sk;θ)}×∏j=1k‍{Λ(tj,sj;θ)-Λ(tj-1,sj-1;θ)}(xj-xj-1)(xj-xj-1)!.
Taking the logarithm of both sides of (17) yields the following logarithmic likelihood function:
(18)lnL(θ)=∑j=1k‍(xj-xj-1)×ln{Λ(tj,sj;θ)-Λ(tj-1,sj-1;θ)}-Λ(tk,sk;θ)-∑j=1k‍ln{(xj-xj-1)!},
where (t0,s0,x0)=(0,0,0).Since the logarithmic function is monotonically increasing, the maximum likelihood estimate of the parameterθmaximizes the logarithmic likelihood function lnL(θ). If the function lnL(θ) is strictly concave in θ, then the maximum likelihood estimate has to satisfy the first order condition of optimality ∂lnL(θ)/∂θ=0, where 0 is the zero vector. The above modeling approach provides a valid parameter estimation procedure based on the maximum likelihood method, without the two-stage procedure based on the least square method.


## 4. Software Release Planning

Of our next concern is the formulation of the optimal software release problem (e.g.,[45–49]) based on the two-dimensional SDMs. The main objective here is to derive simultaneously the optimal software release timing measured by calendar time and the optimal testing-effort expenditure, which minimize the relevant expected software cost. This leads to the determination of the economic software testing schedule. Define: t0:
software release time measured by calendar times0:
cumulative test-execution time measured by CPU timekc:
software-testing cost per unit calendar timeke:
software-testing cost per unit execution timect:
software-fault correction/removal cost per unit fault in testing phaseco:
software-fault correction/removal cost per unit fault in operational phaseLs:
lifetime of software product (given constant).During the testing phase, two kinds of testing costs proportional to the calendar time and the test-execution time, respectively, are considered, wherekct0 is the cost to keep the testing team or the testing personnel, and kes0 is the cost to develop the test cases. Here, it is assumed again that the development cost of test cases is proportional to the time length of software test-execution time. On the other hand, when one software fault is detected in the testing phase and the operational phase, the unit costs to fix/remove it are given by ct and co, respectively. The expected debugging cost during the testing phase is ctΛ(t0,s0) with fixed (t0,s0). If the lifetime of software product Ls is given in advance, then the expected number of software faults detected in the operational phase is given by Λ(Ls,s0)-Λ(t0,s0), that is, the expected debugging cost during the operational phase becomes co{Λ(Ls,s0)-Λ(t0,s0)}. Thus, the expected total software cost during the software life cycle is given by
(19)C(t0,s0)=kct0+kes0+ctΛ(t0,s0)+co{Λ(Ls,s0)-Λ(t0,s0)}.
If Ls→∞ as a special case, then limLs→∞Λ(Ls,s0)=ω and the underlying optimization problem can be simplified as
(20)min(t0,s0):C(t0,s0)=kct0+kes0+ctΛ(t0,s0)+co{ω-Λ(t0,s0)}.For the expected total software cost during the software life cycle, the optimal software release policy is defined by the optimal pair(t0*,s0*) which minimizes it. That is, the pair (t0*,s0*) is the minimizer of C(t0,s0) provided that the associated Hessian matrix is positive definite at (t0*,s0*). In this situation, it must satisfy the first order condition of optimality
(21)∂C(t0,s0)∂t0|t0=t0*=∂C(t0,s0)∂s0|s0=s0*=0
if there exists a unique solution (t0*,s0*). Fortunately, it can be shown that the two-dimensional minimization problem is rather tractable. Figure 3 illustrates the behavior of expected total software cost function with respect to (t0,s0), where kc=100 ($), ke=1 ($), ct=1 ($), co=20 ($), Ls→∞, and M(t0,s0)=ωFMO(t0,s0) with ω=87.5452, a=0.25202, b=0.03984, and c=0.00803. It can be easily seen that the function is unimodal in (t0,s0) and that there exists a unique optimal software release policy (t0*,s0*)=(5.566,85.547). In this simple example, the time length of software testing should be planned with the release time t0=5.566(weeks)≈39(days) and the test-execution time s0=85.542 (CPU hr). When the software delivery schedule is fixed, t0 is replaced by the delivery time, and the underlying problem is reduced to the simpler one-dimensional minimization problem, mins0C(t0,s0) with given t0.Figure 3
Unimodality of the expected total software cost function.


## 5. Real Data Analysis

### 5.1. Data Sets

In this section, we consider two real data sets collected in the actual software development projects for the real time command and control systems [50]. We present these data in Tables 1 and 2, where the data set DS no. 1 (DS no. 2) consists of 54 (53) faults count data observed during 17 (16) weeks with total 32.8 (18.5) (CPU hr). More precisely, DS no. 1 is the fault data for a middle size of program with 27.7 KLC of a real-time command control system. On the other hand, DS no. 2 is also the fault data for the similar program with 33.5 KLC.Table 1
Data set DS no. 1.
t
j
s
j
x
j
t
j
s
j
x
j
1
0.053
1
10
16.200
42
2
0.115
2
11
17.100
42
3
0.273
4
12
18.600
46
4
0.354
5
13
20.600
47
5
1.400
13
14
21.800
47
6
3.150
22
15
23.000
49
7
6.110
28
16
25.200
51
8
11.080
35
17
32.800
54
9
11.500
39Table 2
Data set DS no. 2.
t
j
s
j
x
j
t
j
s
j
x
j
1
0.358
8
9
4.580
48
2
0.690
14
10
6.620
50
3
0.945
17
11
9.980
51
4
1.270
17
12
11.300
51
5
1.670
28
13
13.700
51
6
2.210
35
14
15.300
53
7
2.610
35
15
17.700
53
8
3.760
41
16
18.500
53


### 5.2. Goodness-of-Fit Test

Using these two data sets, we derive the maximum likelihood estimate of the parameterθ, say θ^, and perform the goodness-of-fit test of the two-dimensional SDMs with the maximum logarithmic likelihood (LLF), the Akaike information criterion (AIC) and the residual square sum (RSS) at the data observation points of 100%, 75%, 50%, and 25%. Tables 3 and 4 present the goodness-of-fit results for the data sets DS no. 1 and DS no. 2, respectively, where MO, BW, GO, GL, LT, and WT denote the Marshall-Olkin SDM, the bivariate Weibull SDM in the two-dimensional NHPPs, the one-dimensional exponential SDM, the one-dimensional generalized exponential SDM, the one-dimensional logistic testing-effort dependent SDM, and the one-dimensional Weibull testing-effort dependent SDM, respectively. From these results, it can be seen that the two-dimensional SDMs can provide the better goodness-of-fit performance in terms of AIC and RSS in many cases. Of course, it is worth noting that the comparison between the two-dimensional and the one-dimensional SDMs is not fair, because the corresponding AICs mean the approximate distances in the sense of distribution between the real (but unknown) software fault-count process and the probability model with different dimension. In other words, the goodness-of-fit comparison based on the AIC should be made among the SDMs with the same dimension. Then, the BW can show the better goodness-of-fit performance than the MO in almost cases. In Figures 4 and 5, we show the temporal behavior of estimates for the cumulative number of detected software faults with 100% data of DS no. 1 and DS no. 2, respectively. It is observed that the two-dimensional SDMs clarify the relationship between the calendar time and the testing-effort expenditure and enable measuring the software testing progress effectively.Table 3
Goodness-of-fit test with DS no. 1.
100%
MO
BW
GO
GL
LT
WT
LLF
−45.266
−33.740
−41.468
−41.247
−43.385
−38.518
AIC
98.532
79.480
86.936
88.493
90.769
81.036
RSS
71.160
9.216
25.513
44.988
27.078
15.730
75%
MO
BW
GO
GL
LT
WT
LLF
−33.334
−25.132
−31.540
−24.431
−26.493
−24.842
AIC
74.668
62.263
67.080
54.861
56.986
53.684
RSS
30.450
2.503
24.673
2.605
6.632
1.784
50%
MO
BW
GO
GW
LT
WT
LLF
−13.358
−12.394
−22.547
−15.423
−14.748
−16.038
AIC
34.716
36.787
49.095
36.846
33.497
36.077
RSS
1.887
0.311
50.024
1.928
1.400
1.142
25%
MO
BW
GO
GL
LT
WT
LLF
−4.327
−4.382
−4.577
−4.504
−4.444
−4.412
AIC
16.653
20.765
13.155
15.009
12.889
12.824
RSS
0.010
0.164
0.094
0.043
0.028
0.044Table 4
Goodness-of-fit test with DS no. 2.
100%
MO
BW
GO
GL
LT
WT
LLF
−48.967
−34.314
−39.951
−39.016
−37.885
−42.403
AIC
105.933
80.629
83.901
84.032
79.770
88.807
RSS
47.519
3.626
12.010
7.868
6.158
9.100
75%
MO
BW
GO
GL
LT
WT
LLF
−29.559
−28.191
−34.645
−34.570
−32.730
−38.166
AIC
67.117
68.383
73.291
75.140
69.459
80.331
RSS
4.664
3.558
9.203
8.623
6.390
11.790
50%
MO
BW
GO
GL
LT
WT
LLF
−22.231
−22.083
−25.327
−24.996
−25.606
−27.059
AIC
52.463
56.167
54.653
55.992
55.211
58.118
RSS
4.314
4.708
6.252
6.264
6.626
8.731
25%
MO
BW
GO
GL
LT
WT
LLF
−7.307
−5.493
−6.808
−5.866
−6.895
−7.192
AIC
22.614
22.985
17.615
17.732
17.789
18.384
RSS
0.751
0.261
0.700
0.141
0.745
0.970Estimates of cumulative number of detected software faults (DS no. 1).
(a)
MO
(b)
BW
(c)
One-dimensional SRMsEstimates of cumulative number of detected software faults (DS no. 2).
(a)
MO
(b)
BW
(c)
One-dimensional SRMs


### 5.3. Prediction Performance

Next, we predict the behavior of the number of software faults detected in the future. Two possible cases are considered: in case 1, the model parameters are estimated at the observation pointtn, but at each calendar time t=tn+1,tn+2,…, the testing-effort expenditure is scheduled in advance (s=sn+1,sn+2,…) (given) or estimated with the regression function in (14); that is, s=s^n+1,s^n+2,… (regression). On the other hand, in case 2, we consider the other situation where after estimating the model parameters at the observation point sn, the testing-effort expenditure to consume the scheduled test cases, s=sn+1,sn+2,…, is known in advance, but the calendar time to complete respective test cases is estimated with the regression function in (18), that is, t=t^n+1,t^n+2,…. Tables 5 and 6 (Tables 7 and 8) present the prediction squares error (PSE) for estimates of the cumulative number of software faults detected in the future, where PSE is defined by
(22)PSE=1n'-n∑k=n+1n'‍[xk-Λ(tk,sk;θ^)]2,
and n'(>n) is the total number of data for analysis. In this example, we set n'=17 and 16 for DS no. 1 and DS no. 2, respectively. From these results, the two-dimensional SDMs can show the better performance for prediction at the 75% observation point. However, in the other observation points, no remarkable results are found, so that the prediction performance depends on the kind of data.Table 5
Prediction error in case 1 with DS no. 1.
75%
MO
BW
GO
GL
LT
WT
Given
42.218
1.945
71.224
9.731
16.877
10.880
Regression
7.911
18.681
50%
MO
BW
GO
GL
LT
WT
Given
99.002
103.981
159.551
209.869
119.644
35.762
Regression
120.303
120.310
25%
MO
BW
GO
GL
LT
WT
Given
645.622
1293.560
724.258
1186.030
1277.290
1158.760
Regression
1333.690
1333.450Table 6
Prediction error in case 2 with DS no. 1.
MO
BW
GL
75%
30.661
2.154
4.889
50%
97.368
119.921
111.164
25%
651.142
1333.160
911.021Table 7
Prediction error in case 1 with DS no. 2.
75%
MO
BW
GO
GL
LT
WT
Given
2.431
0.656
10.252
5.825
1.656
0.863
Regression
2.994
0.691
50%
MO
BW
GO
GL
LT
WT
Given
9.812
34.861
77.538
114.273
3.240
4.330
Regression
92.790
107.750
25%
MO
BW
GO
GL
LT
WT
Given
811.770
895.649
842.364
894.170
824.662
795.296
Regression
895.753
677.371Table 8
Prediction error in case 2 with DS no. 2.
MO
BW
GL
75%
2.431
1.844
2.431
50%
9.812
34.861
9.812
25%
811.770
895.890
811.770Next, we assess the quantitative software reliability as the probability that no software failure occurs during a specified time period, where the software reliability is defined by(23)R(x∣tn,sn)=exp[-{Λ(tn+x,sn;θ^)-Λ(tn,sn;θ^)}].
Once the software is released at the calendar time tn to the user or market, it should be noted that no software test is carried out in general, so that the cumulative testing-effort expenditure is considered as a constant, and the software reliability is defined as a function of only the calendar time. Figures 6 and 7 illustrate the estimation results of software reliability at the 100% observation point. From these figures, it can be found that the two-dimensional SDMs (MO and BW) overestimate the reliability function much more than the typical one-dimensional SDMs (note that the testing-effort dependent SDMs are not available in the situation with the constant testing-effort expenditure after release, because W(t) is an increasing function of the calendar time). Since the corresponding software products are actually released at the observation points, however, it can be concluded that the values of software reliability based on one-dimensional SDMs are too small and rather questionable.Figure 6
Estimating software reliability with DS no. 1.Figure 7
Estimating software reliability with DS no. 2.


### 5.4. Optimal Software Release Policy

Finally, we investigate the dependence of cost parameters characterizing the software testing process on the optimal software release policy (Tables9 and 10). In Tables 11 and 12, we present the optimal software release policies maximizing the expected total software costs with the two-dimensional SDMs proposed in this paper, where the Marshall-Olkin SDM is assumed with the same parameters in Figure 3. In these tables, the policy (t0*,s0*)=(0,0) denotes that it is optimal not to carry out the software test economically. Of course, such a situation with relatively small difference between the fault correction/removal costs in the testing and operational phases will not be realistic. As a remarkable point in these tables, as both cost ratios kc/ke and ct/co increase, the associated optimal pair (t0*,s0*) monotonically decrease. To the contrary, the expected total software cost with fixed ct/co increases but the cost with fixed kc/ke decreases.Table 9
Dependence of cost ratiokc/ke on the optimal software release policy with the Marshal and Olkin SRM:  ct=1, co=20.
k
c
/
k
e
(
t
0
*
,
s
0
*
)
C
(
t
0
*
,
s
0
*
)
1/1
(23.284, 91.379)
226.999
5/1
(17.097, 91.155)
304.576
10/1
(14.432, 90.881)
382.635
20/1
(11.766, 90.333)
512.098
50/1
( 8.240, 88.640)
804.229
100/1
( 5.566, 85.542)
1141.750
200/1
( 2.872, 77.581)
1548.600
300/1
( 1.225, 63.745)
1749.270Table 10
Dependence of cost rationct/co on the optimal software release policy with the Marshal and Olkin SRM: kc=100, ke=1.
c
t
/
c
o
(
t
0
*
,
s
0
*
)
C
(
t
0
*
,
s
0
*
)
1/80
(11.098, 119.890)
1724.020
1/50
( 9.253, 109.094)
1529.560
1/30
( 7.219, 96.576)
1315.360
1/20
( 5.566, 85.542)
1141.750
1/15
( 4.357, 76.585)
1015.440
1/10
( 2.539, 60.175)
829.632
1/5
( 0.000, 0.000)
437.726
1/1
( 0.000, 0.000)
87.545Table 11
Dependence of cost ratiokc/ke on the optimal software release policy with the bivariate Weibull SRM: ct=1,co=20.
k
c
/
k
e
(
t
0
*
,
s
0
*
)
C
(
t
0
*
,
s
0
*
)
1/1
(12.280, 139.110)
1133.400
5/1
( 2.462, 136.827)
1153.190
10/1
( 1.232, 135.845)
1161.730
20/1
( 0.616, 134.864)
1170.270
50/1
( 0.247, 133.569)
1181.570
100/1
( 0.123, 132.591)
1190.130
200/1
( 0.062, 131.615)
1198.680
300/1
( 0.041, 131.044)
1203.690Table 12
Dependence of cost ratioct/co on the optimal software release policy with the bivariate Weibull SRM: kc=100, ke=1.
c
t
/
c
o
(
t
0
*
,
s
0
*
)
C
(
t
0
*
,
s
0
*
)
1/80
(0.567, 245.157)
3944.060
1/50
(0.344, 202.974)
2593.110
1/30
(0.196, 161.844)
1668.020
1/20
(0.123, 132.591)
1190.130
1/15
(0.088, 113.591)
943.667
1/10
(0.052,  89.176)
687.952
1/5
(0.019,  53.380)
412.309
1/1
(0.000,   0.000)
120.758


## 5.1. Data Sets

In this section, we consider two real data sets collected in the actual software development projects for the real time command and control systems [50]. We present these data in Tables 1 and 2, where the data set DS no. 1 (DS no. 2) consists of 54 (53) faults count data observed during 17 (16) weeks with total 32.8 (18.5) (CPU hr). More precisely, DS no. 1 is the fault data for a middle size of program with 27.7 KLC of a real-time command control system. On the other hand, DS no. 2 is also the fault data for the similar program with 33.5 KLC.Table 1
Data set DS no. 1.
t
j
s
j
x
j
t
j
s
j
x
j
1
0.053
1
10
16.200
42
2
0.115
2
11
17.100
42
3
0.273
4
12
18.600
46
4
0.354
5
13
20.600
47
5
1.400
13
14
21.800
47
6
3.150
22
15
23.000
49
7
6.110
28
16
25.200
51
8
11.080
35
17
32.800
54
9
11.500
39Table 2
Data set DS no. 2.
t
j
s
j
x
j
t
j
s
j
x
j
1
0.358
8
9
4.580
48
2
0.690
14
10
6.620
50
3
0.945
17
11
9.980
51
4
1.270
17
12
11.300
51
5
1.670
28
13
13.700
51
6
2.210
35
14
15.300
53
7
2.610
35
15
17.700
53
8
3.760
41
16
18.500
53


## 5.2. Goodness-of-Fit Test

Using these two data sets, we derive the maximum likelihood estimate of the parameterθ, say θ^, and perform the goodness-of-fit test of the two-dimensional SDMs with the maximum logarithmic likelihood (LLF), the Akaike information criterion (AIC) and the residual square sum (RSS) at the data observation points of 100%, 75%, 50%, and 25%. Tables 3 and 4 present the goodness-of-fit results for the data sets DS no. 1 and DS no. 2, respectively, where MO, BW, GO, GL, LT, and WT denote the Marshall-Olkin SDM, the bivariate Weibull SDM in the two-dimensional NHPPs, the one-dimensional exponential SDM, the one-dimensional generalized exponential SDM, the one-dimensional logistic testing-effort dependent SDM, and the one-dimensional Weibull testing-effort dependent SDM, respectively. From these results, it can be seen that the two-dimensional SDMs can provide the better goodness-of-fit performance in terms of AIC and RSS in many cases. Of course, it is worth noting that the comparison between the two-dimensional and the one-dimensional SDMs is not fair, because the corresponding AICs mean the approximate distances in the sense of distribution between the real (but unknown) software fault-count process and the probability model with different dimension. In other words, the goodness-of-fit comparison based on the AIC should be made among the SDMs with the same dimension. Then, the BW can show the better goodness-of-fit performance than the MO in almost cases. In Figures 4 and 5, we show the temporal behavior of estimates for the cumulative number of detected software faults with 100% data of DS no. 1 and DS no. 2, respectively. It is observed that the two-dimensional SDMs clarify the relationship between the calendar time and the testing-effort expenditure and enable measuring the software testing progress effectively.Table 3
Goodness-of-fit test with DS no. 1.
100%
MO
BW
GO
GL
LT
WT
LLF
−45.266
−33.740
−41.468
−41.247
−43.385
−38.518
AIC
98.532
79.480
86.936
88.493
90.769
81.036
RSS
71.160
9.216
25.513
44.988
27.078
15.730
75%
MO
BW
GO
GL
LT
WT
LLF
−33.334
−25.132
−31.540
−24.431
−26.493
−24.842
AIC
74.668
62.263
67.080
54.861
56.986
53.684
RSS
30.450
2.503
24.673
2.605
6.632
1.784
50%
MO
BW
GO
GW
LT
WT
LLF
−13.358
−12.394
−22.547
−15.423
−14.748
−16.038
AIC
34.716
36.787
49.095
36.846
33.497
36.077
RSS
1.887
0.311
50.024
1.928
1.400
1.142
25%
MO
BW
GO
GL
LT
WT
LLF
−4.327
−4.382
−4.577
−4.504
−4.444
−4.412
AIC
16.653
20.765
13.155
15.009
12.889
12.824
RSS
0.010
0.164
0.094
0.043
0.028
0.044Table 4
Goodness-of-fit test with DS no. 2.
100%
MO
BW
GO
GL
LT
WT
LLF
−48.967
−34.314
−39.951
−39.016
−37.885
−42.403
AIC
105.933
80.629
83.901
84.032
79.770
88.807
RSS
47.519
3.626
12.010
7.868
6.158
9.100
75%
MO
BW
GO
GL
LT
WT
LLF
−29.559
−28.191
−34.645
−34.570
−32.730
−38.166
AIC
67.117
68.383
73.291
75.140
69.459
80.331
RSS
4.664
3.558
9.203
8.623
6.390
11.790
50%
MO
BW
GO
GL
LT
WT
LLF
−22.231
−22.083
−25.327
−24.996
−25.606
−27.059
AIC
52.463
56.167
54.653
55.992
55.211
58.118
RSS
4.314
4.708
6.252
6.264
6.626
8.731
25%
MO
BW
GO
GL
LT
WT
LLF
−7.307
−5.493
−6.808
−5.866
−6.895
−7.192
AIC
22.614
22.985
17.615
17.732
17.789
18.384
RSS
0.751
0.261
0.700
0.141
0.745
0.970Estimates of cumulative number of detected software faults (DS no. 1).
(a)
MO
(b)
BW
(c)
One-dimensional SRMsEstimates of cumulative number of detected software faults (DS no. 2).
(a)
MO
(b)
BW
(c)
One-dimensional SRMs


## 5.3. Prediction Performance

Next, we predict the behavior of the number of software faults detected in the future. Two possible cases are considered: in case 1, the model parameters are estimated at the observation pointtn, but at each calendar time t=tn+1,tn+2,…, the testing-effort expenditure is scheduled in advance (s=sn+1,sn+2,…) (given) or estimated with the regression function in (14); that is, s=s^n+1,s^n+2,… (regression). On the other hand, in case 2, we consider the other situation where after estimating the model parameters at the observation point sn, the testing-effort expenditure to consume the scheduled test cases, s=sn+1,sn+2,…, is known in advance, but the calendar time to complete respective test cases is estimated with the regression function in (18), that is, t=t^n+1,t^n+2,…. Tables 5 and 6 (Tables 7 and 8) present the prediction squares error (PSE) for estimates of the cumulative number of software faults detected in the future, where PSE is defined by
(22)PSE=1n'-n∑k=n+1n'‍[xk-Λ(tk,sk;θ^)]2,
and n'(>n) is the total number of data for analysis. In this example, we set n'=17 and 16 for DS no. 1 and DS no. 2, respectively. From these results, the two-dimensional SDMs can show the better performance for prediction at the 75% observation point. However, in the other observation points, no remarkable results are found, so that the prediction performance depends on the kind of data.Table 5
Prediction error in case 1 with DS no. 1.
75%
MO
BW
GO
GL
LT
WT
Given
42.218
1.945
71.224
9.731
16.877
10.880
Regression
7.911
18.681
50%
MO
BW
GO
GL
LT
WT
Given
99.002
103.981
159.551
209.869
119.644
35.762
Regression
120.303
120.310
25%
MO
BW
GO
GL
LT
WT
Given
645.622
1293.560
724.258
1186.030
1277.290
1158.760
Regression
1333.690
1333.450Table 6
Prediction error in case 2 with DS no. 1.
MO
BW
GL
75%
30.661
2.154
4.889
50%
97.368
119.921
111.164
25%
651.142
1333.160
911.021Table 7
Prediction error in case 1 with DS no. 2.
75%
MO
BW
GO
GL
LT
WT
Given
2.431
0.656
10.252
5.825
1.656
0.863
Regression
2.994
0.691
50%
MO
BW
GO
GL
LT
WT
Given
9.812
34.861
77.538
114.273
3.240
4.330
Regression
92.790
107.750
25%
MO
BW
GO
GL
LT
WT
Given
811.770
895.649
842.364
894.170
824.662
795.296
Regression
895.753
677.371Table 8
Prediction error in case 2 with DS no. 2.
MO
BW
GL
75%
2.431
1.844
2.431
50%
9.812
34.861
9.812
25%
811.770
895.890
811.770Next, we assess the quantitative software reliability as the probability that no software failure occurs during a specified time period, where the software reliability is defined by(23)R(x∣tn,sn)=exp[-{Λ(tn+x,sn;θ^)-Λ(tn,sn;θ^)}].
Once the software is released at the calendar time tn to the user or market, it should be noted that no software test is carried out in general, so that the cumulative testing-effort expenditure is considered as a constant, and the software reliability is defined as a function of only the calendar time. Figures 6 and 7 illustrate the estimation results of software reliability at the 100% observation point. From these figures, it can be found that the two-dimensional SDMs (MO and BW) overestimate the reliability function much more than the typical one-dimensional SDMs (note that the testing-effort dependent SDMs are not available in the situation with the constant testing-effort expenditure after release, because W(t) is an increasing function of the calendar time). Since the corresponding software products are actually released at the observation points, however, it can be concluded that the values of software reliability based on one-dimensional SDMs are too small and rather questionable.Figure 6
Estimating software reliability with DS no. 1.Figure 7
Estimating software reliability with DS no. 2.


## 5.4. Optimal Software Release Policy

Finally, we investigate the dependence of cost parameters characterizing the software testing process on the optimal software release policy (Tables9 and 10). In Tables 11 and 12, we present the optimal software release policies maximizing the expected total software costs with the two-dimensional SDMs proposed in this paper, where the Marshall-Olkin SDM is assumed with the same parameters in Figure 3. In these tables, the policy (t0*,s0*)=(0,0) denotes that it is optimal not to carry out the software test economically. Of course, such a situation with relatively small difference between the fault correction/removal costs in the testing and operational phases will not be realistic. As a remarkable point in these tables, as both cost ratios kc/ke and ct/co increase, the associated optimal pair (t0*,s0*) monotonically decrease. To the contrary, the expected total software cost with fixed ct/co increases but the cost with fixed kc/ke decreases.Table 9
Dependence of cost ratiokc/ke on the optimal software release policy with the Marshal and Olkin SRM:  ct=1, co=20.
k
c
/
k
e
(
t
0
*
,
s
0
*
)
C
(
t
0
*
,
s
0
*
)
1/1
(23.284, 91.379)
226.999
5/1
(17.097, 91.155)
304.576
10/1
(14.432, 90.881)
382.635
20/1
(11.766, 90.333)
512.098
50/1
( 8.240, 88.640)
804.229
100/1
( 5.566, 85.542)
1141.750
200/1
( 2.872, 77.581)
1548.600
300/1
( 1.225, 63.745)
1749.270Table 10
Dependence of cost rationct/co on the optimal software release policy with the Marshal and Olkin SRM: kc=100, ke=1.
c
t
/
c
o
(
t
0
*
,
s
0
*
)
C
(
t
0
*
,
s
0
*
)
1/80
(11.098, 119.890)
1724.020
1/50
( 9.253, 109.094)
1529.560
1/30
( 7.219, 96.576)
1315.360
1/20
( 5.566, 85.542)
1141.750
1/15
( 4.357, 76.585)
1015.440
1/10
( 2.539, 60.175)
829.632
1/5
( 0.000, 0.000)
437.726
1/1
( 0.000, 0.000)
87.545Table 11
Dependence of cost ratiokc/ke on the optimal software release policy with the bivariate Weibull SRM: ct=1,co=20.
k
c
/
k
e
(
t
0
*
,
s
0
*
)
C
(
t
0
*
,
s
0
*
)
1/1
(12.280, 139.110)
1133.400
5/1
( 2.462, 136.827)
1153.190
10/1
( 1.232, 135.845)
1161.730
20/1
( 0.616, 134.864)
1170.270
50/1
( 0.247, 133.569)
1181.570
100/1
( 0.123, 132.591)
1190.130
200/1
( 0.062, 131.615)
1198.680
300/1
( 0.041, 131.044)
1203.690Table 12
Dependence of cost ratioct/co on the optimal software release policy with the bivariate Weibull SRM: kc=100, ke=1.
c
t
/
c
o
(
t
0
*
,
s
0
*
)
C
(
t
0
*
,
s
0
*
)
1/80
(0.567, 245.157)
3944.060
1/50
(0.344, 202.974)
2593.110
1/30
(0.196, 161.844)
1668.020
1/20
(0.123, 132.591)
1190.130
1/15
(0.088, 113.591)
943.667
1/10
(0.052,  89.176)
687.952
1/5
(0.019,  53.380)
412.309
1/1
(0.000,   0.000)
120.758


## 6. Conclusions

In this paper, we have developed two-dimensional software reliability models based on bivariate fault-detection time distributions related with calendar time and test execution time and compared their goodness-of-fit and prediction performances with real project data. The numerical examples have suggested that our new SDMs provide better performance than the existing NHPP-based SDMs in many cases.In the future, we will apply the same technique to the case with many factors influencing the cumulative number of failure occurrences. For instance, when the code coverage is considered as Grottke [23], a different modeling framework will be needed to incorporate it. Also, if the third time measure is introduced like the number of test cases, the higher-dimensional SDMs should be developed to handle such a complex case.



---

*Source: 413090-2012-11-14.xml*