Estimating the Number of Faults using Simulator based on
Generalized Stochastic Petri-Net Model
Osamu MIZUNO†, Shinji KUSUMOTO†, Tohru KIKUNO†,
Yasunari TAKAGI‡ and Keishi SAKAMOTO‡†
Department of Informatics and Mathematical Science,
Graduate School of Engineering Science, Osaka University‡
Development and Production H.Q., Social Systems Business Group,
OMRON Corporation, Japan
Abstract
In order to manage software projects quantitatively,
we have presented a new model for software projectbased on Generalized Stochastic Petri-net model which
can take inﬂuence of human factors into account, andwe have already developed software project simulator
based on GSPN model. This paper proposes methods
for calculating model parameters in the new model andestimating the number of faults in the design and debug phases of software process. Then we present experimental evaluation of proposed method using a data ofactual software development project on a certain company. As the result of case study, we conﬁrmed its
eﬀectiveness with respect to estimating the number offaults in the software process.

1

Introduction

Increasing the productivity and quality of softwaredevelopment has been an important research objective
in software engineering. It is reported that most companies spend between 50-80% of their development
cost on test process [2]. Thus, putting more emphaseson test process (testing and debugging) has been recognized to be one of the most eﬀective approaches.
Software Reliability Growth Model (SRGM) is oneof the most well-known models for quantitatively evaluating test process [8]. SRGM gives us useful information, e.g., the number of residual faults in the software or MTBF, for deciding the shipping date. On
the other hand, program slicing[10] has been proposed
and applied to localize faults in the program. By usingslicing, debugger can eﬃciently determine the faulty
statements in the program. However, the test process
is very complicated consisting of: planning the general approach, ﬁnding resources and scheduling, determining features to be tested, designing the set oftests, implementing the plan and design, executing thetest procedures, checking for termination, and evaluating the test eﬀort and unit[11]. Thus, only using the
SRGM or slicing is not suﬃcient to improve the test
process.
On the other hands, there are numerous studies
and reports [1][5] regarding the improvement of software development processes, including SEI Capability Maturity Model (CMM) proposed by Humphrey[5]. Based on a process maturity model, SEI SelfAssessment builds a consensus view of an organization’s maturity and the key issues facing the organization. Then ultimately it presents an improvement planfor software development process endorsed by generalmanagement.
In order to attain the improvement of software development process, we have proposed process improvement procedure which describes the software development process using Generalized Stochastic Petrinet(GSPN) and estimates quality, cost and delivery
date of it using the project simulator based on GSPN
[6]. In [6], we showed that the simulator could estimate the value of the development duration and development eﬀort.
In this paper, we show the usefulness of the simulator in estimating the quality of the product (e.g.
the number of residual faults in the ﬁnal product, thenumber of detected faults during reviews and debugs).
That is, by using the simulator with the actual dataof the previous or the past project A and development
plans of the target project B, we propose a method toestimate the quality of the product developed in the
project B.
In the proposed method, at ﬁrst, we determine several parameters in the GSPN using the actual data of
the previous project A. Next, using the development
plans of the target project B, we deﬁne the parameters about workload. Workload is the key factor ofthe proposed model and will be explained in subsection 3.1. Finally, using these parameters, we simulate
the target project B and estimate the numbers of detected and residual faults in each debug activity of the
project B.

2

Preliminary

We assume that software development process consists of two successive phases: design and debug phases(See Figure 1).
As shown in Figure 1, we introduce several parameters to make the discussions clear. Let ei be the number of faults introduced into the product developed indesign/coding activity i. Let dj be the number of detected and removed faults in reviews/debug activity j.
Let ddesign be the total of dCDR , dF DR , dSDR , dM DR
and dP GR . Let rdesign be the number of residual faults
in design phase and is calculated by Σei -ddesign . Also,
let ddebug be the total of dU DB , dIDB , dF DB , anddV DB . Let rdebug be the number of residual faults
in debug phase and is calculated by rdesign -ddebug .
However, as a matter of fact, we cannot collect allvalues of the above parameters. Thus, in this paper,
we aim to estimate the value of ddesign , ddebug andrdebug . In Section 5, we evaluate the usefulness of the
proposed quality estimation method by comparing theestimated values of ddesign , ddebug and rdebug and theactual ones.
Design PhaseeCD

Concept Design(CD)
Concept Design Review
(CDR)

eFD

eSD

Unit Debug
(UDB)

dFDR

Integration Debug
(IDB)

dUDB

dIDB

Function Test
(FT)

d SDR

Function Debug
(FDB)

dFDB

Verification Test
(VT)

dMDR

Verification Debug
(VDB)

d VDB

Coding(PG)
Coding Review
(PGR)

dPGR

ddesign=d CDR+ dFDR+ dSDR+ dMDR+ dPGRr design=e CD+e FD+e SD+e MD+e PG - ddesign

3.1

ddebug=d UDB+d IDB+d FDB+d VDB
r debug=r design - ddebug

Figure 1: Software development process

GSPN Model and Simulator
Workload

In [6], we deﬁne the term “workload” of an activity as the total time needed for a developer who has
the standard capability to complete the activity. Additionally, an eﬃciency of the activity under such a
condition is quantiﬁed as 1. The value of eﬃciency
depends on the environment, such as the number ofthe developers, the necessity of communication andperformance of CASE tools. Then, the development
time is calculated as the result of dividing the workload by the eﬃciency of the activity.
If we get the workload of an activity, then we canestimate the development time appropriate for speciﬁc several activity conditions dependent on a given
environment.
In the proposed model, the workload is assigned
to each activity depending on the input products forthe activity. That is, for example, workload of design
activity (Wdesign ) is deﬁned as the following formula:Wdesign = sdesign × Rdesign .
Here, sdesign denotes the size of input product of design activity and Rdesign denotes the workload parameter for design activity. Before simulation, workload
parameter must be given to each activity of the target
project.
Consuming of the workload assigned to an activity corresponds to the progress of the activity in thedevelopment. Growth of product can be modeled bychanging values of the size or the number of faults inthe output product.

3.2

Integration Test
(IT)

ModuleDesign(MD)
Module Design Review
(MDR)

ePG

dCDR

Structure Design(SD)
Structure Design Review
(SDR)

eMD

Unit Test(UT)

Function Design(FD)
FunctionDesign Review(FDR)

Debug Phase3

Outline of the Model

The proposed model consists of Project model andProcess model. Figure 2 shows the outline of the proposed model.
Project model includes three key components: activities, products and developers. Some attributes areattached to each of them, as shown in Tables 1.
Process model includes a set of Activity models
which include speciﬁcations of design, coding, review,
test, debug activities, and so on. Activity model is described by an extended GSPN, as shown in Figure 3.

3.3

Project Model

Project model focuses on three key components: activities, products and developers, and attaches several
attributes to each of them (See Table 1).
An activity has eight kinds of attributes, which
are type, entry/exit conditions, input/output products,
workforce, deadline and workload. (1) Type shows
which the activity corresponds to and describes currently one of design, coding, review, test and debug.

Project model
activities{A i}developers{M i}products{P i}

attribute valuesof Ai Pi Mi

firing rates oftransitions in GSPN
designtest
codingreviewdebug

to the activity and consumed amount of it.
A product has three kinds of attributes, which aresize, the number of faults and completion rate. (1)Size represents the product size in document pages or
the lines of source code. (2) Number of faults counts
faults in the product. (3) Completion rate represents
the ratio of the consumed workload to the assigned
workload.
A developer has an attribute experience level which
is determined according to his/her length of service.
We classify developers’ experience levels into the following three levels: novice, standard and expert levels.
They are quantiﬁed as discrete values 1, 2 and 3, respectively.

3.4
Process model

Figure 2: Outline of the proposed model
Table 1: Project template
Attributes of activity Ai
type
entry condition
exit condition
input products
output products
workforce
deadline
workload
Attributes of product Pi
size
number of faults
completion rate
Attribute of developer Mi
experience level

Activity Model for Process

Activity model is prepared for each type of activities such as design, coding, review, test, debug andso on. The descriptions of Activity models are given
using an extended GSPN. Figure 3 shows an exampleof the description of the design activity. In the extended GSPN, a token has three attributes: productsize s, number of faults f and consumed workload w.
These attributes are used to represent the current status of development that varies over the execution ofeach Activity model.
attributes of tokens product size
f number of faults
w consumed workloadt 1 rcmrt 2 thP2

P1(2) Entry condition and (3) exit condition specify conditions for beginning and ending the activity, respectively. (4) Input products describes the products given
to the activity as the input products and the workload
parameter, that is the degree of contribution of each
input products to determine workload of the activity. (5) Output products describes the output productsthat are developed in the activity and the weight assigned to each product. The variation of the productsize and that of the number of faults are distributed tothe products according to weights. Thus, the sum ofeach weight must be one. (6) Workforce speciﬁes tuples of the developers who engage in the activity andthe ratio of time in which each developer can engage in
the activity to his or her business hours. (7) Deadline
represents the appointed date for the completion of theactivity, which is ﬁxed on the development plan. (8)Workload represents a tuple of the workload assigned

transition
executionfunctiont 3 rwr

t1
-

t2

t3

s=s+1
w=w+1
f=f+1 (pin )Figure 3: Activity modelTransitions used here are timed transitions. The
ﬁring delay of each transition is exponentially distributed and the average ﬁring delay of a transitionis speciﬁed by a ﬁring rate assigned to it. In Figure 3,
the ﬁring rate rcm of transition t1 means that the average ﬁring delay of transition t1 is 1/rcm .
In addition, each transition has a function (called
execution function) to be evaluated on its ﬁring. The
execution of the function updates the attribute values of the token. Intuitively speaking, each transition
corresponds to the developers’ behavior such as thinking, writing and communicating or an event which oc-

curs during execution of activity. Places correspond to
waiting states for occurrences of behaviors or events.
The ﬁring rates of the transitions are formulatedby the following ten functions fcm , fth , fwr , fpr , frd ,fdt , fmd , fps , flc and fin . These functions should beconcretely speciﬁed based on the property of the target
project.
In the following, M is the number of the developers engage in the activity, L is developer’s experiencelevel, ΣL is the sum of each developer’s experience, S
is the total size of the input products, R is the completion rate of the input products, F is the number offaults of the input products and D is the number of thedays from the current date to the deadline of the activity. Activity model parameters (e.g., Kcm , Kth , Kwr ,and so on) are given to each activity and concernedwith the developers’ behavior. For example, Kcm , Kth
and Kwr correspond to communicating, thinking andwriting respectively.
(1) Communicating rate: rcm = fcm (Kcm , M, ΣL, R)(2) Thinking rate: rth = fth (Kth , M, ΣL)(3) Writing rate: rwr = fwr (Kwr , M, ΣL)(4) Preparing rate: rpr = fpr (Kpr , M, ΣL, S)(5) Reading rate: rrd = frd (Krd , M, ΣL)(6) Fault detecting rate: rdt = fdt (Kdt , M, ΣL, S, F )(7) Fault modifying rate: rmd = fmd (Kmd , M, ΣL)(8) Testcase passing rate: rps = fps (Kps , M )(9) Fault localizing rate: rlc = flc (Klc , M, ΣL, S, F )These make it possible to dynamically determine
the frequency of communications or the diﬃculty inthinking and writing according to the number of developers, experience levels of developers and/or completion rates of input products.
Moreover, the increase of product size s at every
ﬁring of writing transition t3 and the consumption ofworkload at every ﬁring of thinking transition t2 aredescribed by the corresponding execution functions.
At each ﬁring of the transition, the values of token’s
attributes can be changed by evaluating its executionfunction.
Activity model handles fault injections in the design
activity as the stochastic events whose occurrences depend on the fault injection rate pin . In general, pin isformulated by the following function:(10) Fault injection rate:
pin = fin (Kin , M, ΣL, D, R)

By using this function, it is possible to take accountof dynamic inﬂuence on the fault injection rate causedby the stress from deadline of the activity or developers’ experience levels.
[Example 1] In the design Activity model depicted
in Figure 3, for example, transitions t1 and t2 which
represent communicating and thinking behavior, respectively, are enabled to ﬁre when a token exists inthe place P1 . If the communicating transition t1 ﬁres,
it has no eﬀect on the attributes values, and the token returns to the place P1 and only time elapses by
the ﬁring delay. On the other hand, if the transition
t2 ﬁres by evaluating its execution function, then consumed workload w is increased by one, and the token
moves to the place P2 . When the token exists in theplace P2 , only the transition t3 which represents writing behavior is enabled. If the transition t3 ﬁres, thenproduct size s is increased by one, and the number offaults f could be increased according to the fault injection rate pin . After the ﬁring of t3 , the token moves
back to the place P1 .

3.5

Simulator

We have designed and implemented a simulator
which supports description of the target process, executes the process described by Activity model andanalyses the simulation results statistically.
The system consists of ﬁve functional units: projectcontrol unit, activity simulator, user interface unit,
display unit and editor.
Simulations proceed at the intervals of unit time1 .
At ﬁrst, project control unit determines activities
to be executed, based on the current status of the
progress and entry/exit conditions of each activities.
Next, for each executable activity, project control unit
delivers the parameters to activity simulator and directs it to execute activities for a day. Then, activity
simulator executes all of the activities, which are directed to execute by project control unit, using given
parameters and extended GSPN. The execution of anactivity is expressed by the consumption of its workload. When an activity consumes all of assigned workload, the activity is regarded to be completed.
The execution of simulation is able to be suspended
or restarted at any time. Moreover, at every unit timeof simulation, intermediate simulation results can be
stored in the simulation database. The intermediate
simulation results are stored in the same format of
the original project description. Thus, it is possible torestart the simulation using the intermediate simulation results as the input. Also it is possible to change
1 Currently, one unit time is a day (8 hours).

the values of parameters(attributes of project) at any
time of the simulation. For example, we can modify
the number of developers at any time of the simulationand simulate it immediately.

5.1

4(1)Development eﬀort of the projects A and B are
152 (man-days) and 62 (man-days), respectively.

Estimation Methods

Here, we propose two estimation methods for software faults. In the proposed methods, we consider two
similar projects: the previous or the past project A(which has already been completed) and the target
project B.

4.1

Method M 1

In M 1, the input data are actual development dataof the previous project A (called DA ) and the development plan of the target project B (called PB ).
Here, PB consists of the estimated period of each activity and the assignment of developers. The output
data are the values of quality metrics (ddesign , ddebug
and rdebug ) at the design and debug phases in the
project B.
M 1 consists of the following four steps:
Step 1: Using DA , calculate the activity model parameters.
Step 2: Using PB , calculate the workload parametersin design and debug phases.
Step 3: Simulate the design and debug phases.

(2)The size of the system of the projects A and B areabout 12.3 (Kstep) and 6.9 (Ksteps), respectively.
(3)Project members are almost unchanged throughall projects.
(4)Each project uses a standard waterfall model.

5.2

Step 1: Using DA , calculate the activity model parameters.
Step 2: Using DB , calculate the workload parametersin design phase. Also, using PB , calculate theworkload parameters in debug phase.
Step 3: Simulate the design and debug phases.
Step 4: Output the values of quality metrics.

5

rcm = Kcm ×

Method M 2

In M 2, the input data are actual development dataof the previous project A (called DA ), actual development data of design phase of the target project
B(called DB ) and the development plan of the project
B (called PB ). The output data are the values of quality metrics (ddesign , ddebug and rdebug ) at the designand debug phases in the project B.
M 2 consists of the following four steps:

Experimental Evaluation

In order to evaluate the usefulness of the proposed
method, we apply the simulator to two similar software development projects A and B in an organization.

Assumptions

Here, we formulate each ﬁring rate and fault injection rate in the design and coding Activity models. In
the following formulas, M is the number of the developers engaged in the activity, ΣL is the sum of eachdeveloper’s experience level, R is the completion rateof the input products, D is the number of the days
from the current date to the deadline of the activity.
Kcm , Kth , Kwr and Kin are parameters given to eachactivity and concerned with communicating, thinking,writing and fault injection rate, respectively. On deciding these formula, we referred to past works suchas [3][4][7][9].

Step 4: Output the values of quality metrics.

4.2

Characteristics of Target Projects

The main characteristics of the projects are summarized as follows:

M2
ΣL × RΣL
× M = Kth × ΣLM
ΣL
rwr = Kwr ×× M = Kwr × ΣLMM
pin = Kin ××M
ΣL × R × D
rth = Kth ×

As spaces are limited, we omit the description of
Project model, formulas of ﬁring rates, and parameters used by other activity models.

5.3

Preliminary Experiment(Step 1 of M 1
and M 2)

Here, we execute the Step 1 of M 1 and M 2. Thatis, using DA , calculate the activity model parameters. As described in Section 2, we assume thetarget process is shown in Figure 1. Actually, we
have the data shown in Table 2. Thus, in order todetermine the activity model parameters and workload parameters of the project A, we use all of di(i = CDR, F DR, SDR, M DR, P GR, design).
From now on, by using the concept design activity,
we explain the way to determine the activity modelparameters. Before determining the activity model
parameter, we have to determine the workload parameter. For the workload parameter, we use the formula
shown in subsection 3.1. The actual value of the input product size of concept design activity is 50. The
actual value of workload (the total amount of concept
design time) is 121.5. Thus, workload parameter ofconcept design activity is RCD = 121.5/50 = 2.43.
For the activity model parameters of concept design, we determine the values of Kcm , Kth , Kwr andKin . We assign the suitable values at ﬁrst and, then,changed the values so that the simulated results of theconcept design activity at the project A become thesame as the actual data of it.
Similarly, we obtained the workload parameters
and the activity model parameters for all other activities of the project A. The resultant values of theactivity model parameters are shown in Table 3. Concerning the values of the workload parameters we omit
them, since they are nearly to secrets of the organization.
As a matter of course, the simulated values ofddesign (=89) and ddebug (=32.1) are quite close to theactual ddesign (=89) and ddebug (=30) of the project A.

5.5

Experiment 2 (Step 2 through 4 of M2)
Now, we execute the Step 2 through 4 of M 2 toreduce the diﬀerence of ddesign ,ddebug and rdebug . At
Step 2, using DB , we calculate the workload parameters in design phases. As for the workload parametersin debug phase, we use the same one in Step 2 in Experiment 1. The updated values in Experiment 2 areshown in Table 4.
Then, in Steps 3 and 4, we simulate the project B
and get the output values of quality metrics. The
simulation results of the project B are shown in Experiment 2 of Table 5. The simulated values ofddesign , ddebug and rdebug are 17.9, 15.8 and 0.7, respectively. These values are quite close to the actual
values of ddesign (=19), ddebug (=16) and rdebug (=1) of
the project B.
Thus, with respect to the experiments, we can conclude that the proposed method can correctly estimate
the numbers of faults in the design and debug phases.
Table 5: Result of experiments

Real data
Experiment 1
Experiment 2

Table 2: Real data of detected faults
dCDR
23

5.4

dF DR
21
dU DB
3

dSDR
16
dIDB
12

dM DR
12
dF DB
6

dP GR
15
dV DB
9

ddesign89
ddebug30

Experiment 1 (Step 2 through 4 of M1)
Next, we execute the Step 2 through 4 of M 1. At
Step 2, using PB , we calculate the workload parameters in design and debug phases. Basically, we use thesame way described in subsection 5.3 except using thevalues of workload and the input product size in theplan PB instead of the actual values of workload andthe input product size.
As the result, we obtained the values of the workload parameters, shown in Table 4, for all of the activities of the project B.
Then, in Steps 3 and 4, we simulate the project B
and get the output values of quality metrics. The simulation results of the project B are shown in Experiment 1 of Table 5. The simulated values of ddesign ,ddebug and rdebug are 23.9, 14.9 and 1.1, respectively.
Though the simulated values of ddebug and rdebug areclose to the actual ddebug (=16) and rdebug (=1) of the
project B, there is a relatively large diﬀerence between
the simulated value of ddesign and the actual one.

6

ddesign

ddebugrdebug

19
23.9
17.9

16
14.9
15.8

1
1.1
0.7

Conclusion

In this paper, we applied the simulator to theevaluation of the testing plan. In our proposed
method, we determined the model parameter of GSPN
model using the development data of the previous
project A. Next, we determined the workload parameter of project model using the development plan of the
project B, which is the target project. Then, we simulated the project B using the above parameters andestimated the number of detected and residual faults
in design and test phases of the project B.
Finally, we experimentally evaluated the proposed
method using the data of actual software development
project on a certain company. Though it has some
limitations, we conﬁrmed the eﬀectiveness of the proposed method.

References[1] V. R. Basili and H. D. Rombach: “TheTAME project: Towards improvement-oriented
software environment,” IEEE Transactions on
Software Engineering, Vol.14, No.6, pp.758-773(1988).
[2] J. S. Collofello and S. N. Woodﬁeld: “Evaluating the eﬀectiveness of reliability-assuranceTable 3: Values of activity model parameters
designcodingreview
test
debug

Kcm

Kth

Kpr

Klc

Krd

Kps

Kdt

Kwr

Kmd

Kin

0.10
0.10
0.08
0.10
0.08

0.20
0.20
—
—
—

—
—
0.10
0.20
—

—
—
—
—
0.12

—
—
3.00
—
—

—
—
—
6.00
—

—
—
0.35
0.24
—

0.20
0.20
—
0.20
—

—
—
0.20
—
0.10

15.5
17.0
—
—
—

Table 4: Values of workload parameters
Experiment 1
Experiment 2

RCD

RCDR

RF D

RF DR

···

RV DB

1.14
1.88

0.45
0.30

4.20
1.17

0.5
1.0

···
···

2.0
2.0

techniques,” Journal of Systems & Software,
Vol.9, No.3, pp.191-195 (1989).
[3] B. Curtis, H. Krasner and N. Iscoe: “A ﬁeld
study of the software design process for large
systems”, Communications of the ACM, Vol.31,
No.11, pp.1268-1287 (1988).
[4] T. Furuyama, Y. Arai and K. Iio: “Fault generation model and mental stress eﬀect analysis”,
The Journal of Systems and Software, Vol.26,
pp.31-42 (1994).
[5] W. S. Humphrey: “Characterizing the software
process: A maturity framework,” IEEE Software, Vol.5, No.2, pp.73-79 (1988).
[6] S. Kusumoto, O. Mizuno, T. Kikuno, Y. Takagi and K. Sakamoto:“A new software project
simulator based on generalized stochastic petrinet”, Proc. of the 19th International Conference
on Software Engineering, pp.293-302 (1997).
[7] K. Matsumoto, S. Kusumoto, T. Kikuno andK. Torii: “An experimental evaluation of teamperformance in program development basedon model – Extension of programmer performance model”, Journal of Information Processing, Vol.15, No.3, pp.466-473 (1992) (inJapanese).
[8] J.D. Musa, A. Iannino and K. Okumoto: Software Reliability: Measurement, Prediction, Application, McGraw-Hill (1987).
[9] H. Sackman, W. J. Erickson and E. E. Grant:“Exploratory experimental studies comparing
online and oﬄine programming performance”,
Communications of the ACM, Vol.11, No.1,
pp.3-11 (1968).

[10] M. Weiser: “Programmers use slices when debugging”, Communications of the ACM, Vol.25,
No.7, pp 446-452 (1982).
[11] “IEEE Standard for Unit Testing”, IEEE. Rep.
IEEE-Std-1008-1987 (1987).