IEICE TRANS. FUNDAMENTALS, VOL.E82{??, NO.1 JANUARY 1999

1

INVITED PAPER Special Section on Concurrent Systems Technology

Enhancing Software Project Simulator toward Risk
Prediction with Cost Estimation Capability
Member, Daisuke SHIMODAy , Nonmember,
Member, and Yasunari TAKAGIyy , Nonmember

Osamu MIZUNOy a) ,Tohru KIKUNOy c) ,This paper presents an enhancement of a software project simulator to perform risk prediction with cost estimation capability.
So far, we have developed a software project simulator tosimulate software development projects. In this simulator, a development process was described using Petri net model, and it
was applied to some actual project data in a certain company
successfully. On the other hand, we have also presented a risk
predicting system to nd \risky" projects by statistical analysis
on risk questionnaire for project managers. In this approach, only
the probability to be risky was calculated for a project. Thus,
the managers in the company wanted to know a concrete proof
why a software project becomes risky.
In this paper, to present the proof that a software project
becomes risky, we try to enhance the previous project simulator
so that the simulator can deal with risk factors. To consider
the risk factors, we modify the previous simulator so that boththe uctuation of skill level and the deadline pressure can be
represented by the parameters in the simulator. By using a casestudy, we con rm that the enhanced simulator can estimate thedevelopment cost under some typical risks. As a result, we can
expect that the simulator shows how much the development costof a risky project exceeds an estimate.
SUMMARY

key words:

Software development project, Simulator, Risk fac-

tors, Cost estimation

1.

Introduction

Generally, a software development process includes concurrent execution of many activities such as design,coding, review, test, and debug. Thus, as software
processes recently have become huge and complex, themanagement of software projects is diÆcult. As a result, various kinds of managemental problems such ascost excess, late delivery date, and low quality, havealso become critical in industry. In order to solve suchproblems, software process improvement has been extensively performed[1].
In a certain company, the Software Engineering
Process Group (SEPG) was established 8 years ago.
The SEPG has tackled the process improvement activManuscript received March 30, 2001.
Manuscript revised June 4, 2001.
y
The authors are with Dept. of Informatics and Mathematical Science, Graduate School of Engineering Science,
Osaka University.
yy
The author is with OMRON Corporation.
a) E-mail: o-mizuno@ics.es.osaka-u.ac.jpb) E-mail: d-simoda@ics.es.osaka-u.ac.jpc) E-mail: kikuno@ics.es.osaka-u.ac.jpd) E-mail: taka@biwa.kusatsu.omron.co.jpb)d)ities in the company so far. We have already performed
several joint works with the SEPG for software process
improvement.
One of our successful joint works with the SEPG
was the formal modeling of software processes by a Generalized Stochastic Petri Net(GSPN). Furthermore, inorder to estimate the quality, cost, and duration of
the software development process, we have developeda software project simulator based on the GSPN[2],[3]. The detailed parameters in the simulator were determined based on experience and data from softwaredevelopment processes in the company. Several experiments showed that software projects were simulated
rather accurately according to the initial developmentplan.
On the other hand, we have also performed statistical analysis of project data collected from a numberof projects. During the analysis, we have noticed that
there were projects that appeared to be out of control
from the project managers' viewpoint. We called suchprojects \risky projects." We then tried to develop arisk predicting system, which calculates the probability based on risk questionnaire for project managers
and nds such risky projects according to the probability. An experimental application showed that most of
the risky projects can be detected by the probability[4].
The project managers were, however, still skeptical because the probability does not provide a concrete proof
why projects become risky. The managers in the company wanted to know such a concrete proof.
In this paper, we try to estimate development
costs, which are an important metric to determine risky
projects. To do so, the project simulator must havecapability to deal with risk factors in the risk questionnaire and to estimate development costs.
For the previous simulator, it is impossible to simulate a project under risks, because the previous simulator cannot represent the risk factors. In the enhanced
simulator, we implement a mechanism that adjusts parameters to deal with the in uence of the risk factors.
As a result, the following typical confusions of risky
projects can be represented in the simulator: confusioncaused by the uctuation of the developer's skill level
and the confusion caused by the deadline pressure[5].
Finally, we perform a case study to con rm
whether a risky project can be simulated. The results

IEICE TRANS. FUNDAMENTALS, VOL.E82{??, NO.1 JANUARY 1999

2

show that the enhanced simulator can estimate the development costs for both ideal case and risky case. As
a result, we can expect that the simulator shows how
much the development cost of a risky project exceeds
an estimate.

twr

rcm

rwr

tth

tcm

rth

2.

Previous Works [2][4]The Software Engineering Process Group (SEPG) has
performed various activities to improve the software
process in the company. We have cooperated with the
SEPG and have made several empirical studies[2], [4].
2.1 Software project simulator

tin
tcm

tth

twr

Meaning

Communication

Think

Write

Executionfunction

-

w=w+1

s=s+1

Attributes of a productsize
number of faults
completion rate
Attributes of a developer
skill level
Fig. 1

Template of Project model

A Project model describes the control ow of thecomponents included in a project as shown in Figure 3.
It also speci es resources and conditions for a projectsuch as the assignment of developers, the schedule ofdevelopment, and so on. An Activity model describes
how to calculate development cost, duration, and thenumber of faults for each activity. Process model describes how to execute Activity models according to thespeci ed conditions.
The Process model can take the in uence of human factors into account by introducing the conceptof \workload." The workload of an activity is de ned
as the total time needed for a developer with averagecapability to complete the activity. The workload canre ect the communication among the developers[6], theperformance of CASE tools, and so on, and thus thetin
Write witha fault
s=s+1
f=f+1

s : size of productw : consumed workload
f : number of faults
Fig. 2In order to estimate the quality and quantity of softwaredevelopment activities, we have developed a software
project simulator based on Generalized Stochastic PetriNets(GSPNs)[2]. The simulator consists of a Projectmodel and a Process model. The Project model focuses
3 key components: activities, products, and developers(See Figure 1). The Process model includes a set of
Activity models (See Figure 2), each of which speci es
process activities in a software development.
Attributes of an activity
type
entry condition
exit condition
input products
output products
deadline
workloaddevelopers

rin

Fig. 3

Design Activity model

Control ow of components

simulator can evaluate the dynamic aspects of software
projects[7].
The Activity models in a Process model are described by GSPNs. The Activity model is prepared foreach activity in a software development process, suchas design, coding, review, test, and debug. An example
of an Activity model for the design activity is shown in
Figure 2.
Each transition of Figure 2 denotes the behavior ofa developer in the activity. Transition tcm represents
the communication among developers, tth represents
the thinking of a developer, twr represents the writing of a developer without faults, and tin represents
the writing with injecting a fault. The ring delay ofeach transition is exponentially distributed and the ring rate of a transition is assigned as r . For example,
the ring rate rcm = 0:2 of transition tcm means thatthe average ring delay of transition tcm is 1=rcm = 5.
Each ring rate is determined by the functions suchas rcm = f (M; L; R) where M is the number of developers, L is experience level of developers, and R is acompletion rate of an input product.
The token in the Activity model has some at-MIZUNO et al.: ENHANCING SOFTWARE PROJECT SIMULATOR TOWARD RISK PREDICTION

tributes such as \consumed workload (w)," \number of
injected faults (f )," and \size of output product (s)."When a transition is red, a function attached to thetransition is executed to change the attributes of thetoken. For example, in Figure 2, function w = w + 1
denotes the consumption of the workload.
The detail of the de nitions of activities are described in the reference [8]. The prototype of the simulator was developed and several experiments have been
done to show the e ectiveness of the proposed simulator[3].
2.2 Risk predicting system
On the other hand, we have tried to predict the nal
status of software development projects by statisticalmethods[4]. The proposed method was based on a questionnaire for the risk factors in software development.
Table 1Requirements

Project Plan

Original questionnaire used in [4]Risk factors
Required performance
Technical difficulty
Requirements specification review
Number of requirements changes
Resultant products in the project plan
Review plan in the project plan
Project plan review
Project plan review by the manager

Optimism for the technical issues
Appropriate estimation
Estimation
Number of projects referred
Needed time to make estimate
Unclear project management method
Possibility of size overcome
Possibility of cost overcome
Project
Possibility of duration overcome
Taking track of the progress
Management
Frequency of progress report
Unresolved problems
Review effort ratio
The morale of developers
Number of leaders in the project
Number of developers
Average working time of developers
Developers
Organization chart
Experience of the requirement describer
Average experience of the developer
Responsive person for each activity in the WBS
Number of group meetings
Communication Number of meetings with managers
Number of meetings with external groups
Untechnical pressure
External Risks Number of COTS packages
Number of external companies cooperated with

Evaluation
High 3 - 0 Low
Difficult 3 - 0 Easy
Done | Not#
Described | Not
Described | Not
Done | Not
Done | Not
Exists | Not
Done | Not#
Days
Unclear 3 - 0 Clear
High 3 - 0 Low
High 3 - 0 Low
High 3 - 0 Low
Yes | No#/Month#%Strong 3 - 0 Weak##
Hour/Day
Made | Not
High 3 - 0 Low
High 3 - 0 Low
Specified | Not#/Month#/Month#/MonthExists | Not##

3(For example, for evaluations `High', `Relatively high',`Relatively low', and `Low', the values 3, 2, 1, and 0 are
assigned, respectively), and the absolute value (such as
the number of requirements changes).
Based on the responses to the questionnaire, wecollected risk assessment data and applied the following
logistic model to them:

P (Y jx1 ;    ; x ) =ne 0 + 1 1 ++ n n
1 + e 0 + 1 1 ++ n n
b

b x

b

b

b x

x
b

x

where x1 ;    ; xn are explanatory variables in themodel, and Y is a binary dependent variable that represents whether a project is risky or not. P is the conditional probability that Y = 1(i.e. a project is risky)when the values of x1 ;    ; xn are determined. We select the risk factors in the questionnaire as potential
explanatory variables, and estimate the coeÆcients bi 's
using the risk assessment data obtained from the responses to the questionnaire. The following is a logistic
model constructed for the risk data at a certain company[4]:P (Y jx1 ; x2 ) =e 5 251+2 727 1 +3 984 2
1 + e 5 251+2 727 1 +3 984 2
:

:

:

x

:

:

x

x

:

x

where x1 and x2 are the risk factors \Estimation" and\Project Plan," respectivelyyy .
We carried out an e ectiveness analysis of the constructed model. The result showed that the constructed
model can nicely predict risky projects based on theprobability P for new data sets.
We also developed a WWW based system that
distributes questionnaires to the developers, helps thestatistical analysis by the SEPG, and predicts the nal status of projects by applying the logistic model tothe projects. The prototype of the predicting system
has been developed, and applied to actual development
projects in the company.
3.

Needs for Enhancement

3.1 Cost estimation of risky projectFirst, we de ned a \risky project" from the viewpoint of development cost and duration. We then designed the questionnaire that includes risk factors tooccur in software development. We designed a questionnaire shown in Table 1 to be distributed to project
managers of software projectsy . The questionnaire includes 34 risk factors which are classi ed into 7 major categories: Requirements, Project plan, Estimation,
Project management, Developers, Communication, andExternal risks. Then project managers return evaluations to each risk factor as shown in Table 1. The
evaluations have the following three types: the binary
evaluation (such as `Yes' or `No'), the ordinal evaluation

As mentioned in subsection 2.2, the prediction of thenal status of a project (risky or not) using the probability was successfully applied to actual projects. However, the project managers were still skeptical because
they were provided with only the probabilities of risk.
They actually wanted to know exactly what will happen in such risky projects.
We therefore tried to provide more concrete proofof risky projects. In more detail, we estimate the development cost of such risky projects, since the development cost is one of the important criteria to determine
risky projects.

yTo be exact, the questionnaire shown in Table 1 isslightly extended from that in reference [4].

yyIn [4], the risk factors were chosen from the categories
in Table 1.

IEICE TRANS. FUNDAMENTALS, VOL.E82{??, NO.1 JANUARY 19994

One of the possible ways to estimate the cost is
to execute the project simulator under software risks.
However, the previous simulator developed in [2] did
not take the in uence of software risks into account.
Thus, in order to estimate the cost of risky projects,
we enhanced the project simulator so that the softwarerisks can be considered.
3.2 An approach for cost estimation
As a result of our investigation, we found that the following two confusions, which are closely related to therisk factors in Table 1, will have a major e ect on thecost.
Confusion caused by a risky project: In a risky
project, developers cannot bring their ability intofull play[9]. That is, in a risky project, the skill
level of a developer is reduced from his/her original level. It is known that the skill level of a developer signi cantly in uences the cost of software
projects[10], [11]. In the enhanced simulator, thedeveloper's skill level can uctuate according to
the risks.
Confusion caused by deadline pressure: The deadline pressure causes many injected faults[5], [12],and thus increases the cost of the latter phasesof a project. In a risky project, since schedule isfrequently delayed[9], the in uence of the deadline
pressure increases greatly. If the development duration of an activity is 90% completed and the assigned workload is not completed 90%, then thedevelopers feel deadline pressure. In the enhanced
simulator, the deadline pressure is represented asa parameter in the fault injection rate.
As will be described in the next section, these situations are implemented by adjusting the parameters in
the project simulator according to evaluation results
for risk factors. As a result, we expect that the cost ofrisky projects can be estimated more correctly.
4.

Enhancement of Project Simulator

In this section, we try to enhance the previous simulator
so that the simulator can deal with the risk factors.
4.1 Selected risk factors
From Table 1, we selected risk factors to be added tothe simulator. First, we identi ed and deleted 16 risk
factors for which there was insuÆcient data for analysis.
However, since important risk factors may be includedin these deleted factors, we consider further analysis ofthem as an important future work. We then deleted 3
risk factors that can be represented directly as an initial
input for the simulator. As a result, 15 risk factors
in Table 2 remained to be selected and added to theenhanced simulator.

Risk factors to be used in enhanced simulator

Table 2Requirements
Project Plan

Estimation
Project
Management
Developers
External Risks

Risk factors
Requirements specification review
Number of requirements changes
Resultant products in the project plan
Project plan review
Project plan review by the manager
Optimism for the technical issues
Appropriate estimation
Needed time to make estimate
Unclear project management method
Review effort ratio
The morale of developers
Experience of the requirement describer
Average experience of the developer
Responsive person for each activity in the WBS
Untechnical pressure

Evaluation
Done | Not#
Described | Not
Done | Not
Done | Not
Exists | Not
Done | Not
Days
Unclear 3 - 0 Clear%
Strong 3 - 0 Weak
High 3 - 0 Low
High 3 - 0 Low
Specified | Not
Exists | Not

4.2 Parameters to represent the risk factors
In order to represent the risk factors in Table 2, we used
the parameters in the project simulator. In the project
simulator, the following parameters are used in ringrate functions: The skill level of developers (L) The mental stress caused by deadline pressure (D) The completion rate of the input products (R) The number of faults in the input products (F ) The number of developers in an activity (M ) The constants for each developer's behavior (K ) The assigned workload by a development plan
(W L)

For example, the ring rate functions transitions in Figure 2 are as followsy :M  (M 1)
rcm = Kcm 

LR

r =K
th

r

wr

th

= Kwr  L  D

r =K
in

 DL

in

 RF +D1

The initial values of these parameters must be
given in a project description. An example is shownin Figure 4. It shows the speci ed values for the activities such as FD, FDR, PG, and so on (As shown in
Figure 6). The number of developers(M ) for Activity 0
is 2, the initial skill levels(L) of all developers in thisproject are 1, the completion rate(R) of Product 0 is
1.0, and the workload(W L) for the Activity 0 is 128.
In the enhanced simulator, the values of parameters are adjusted according to the given software risks.
(The details for the adjusting parameters will be explained in Section 5.) In the following subsections 4.3
and 4.4, we will explain how the risky situations in subsection 3.2 are represented by adjusting the parameters.
y
The de nitions of ring rate functions are slightly extended from that of reference [2].

MIZUNO et al.: ENHANCING SOFTWARE PROJECT SIMULATOR TOWARD RISK PREDICTION

5

Activities’ Definition
Injected Detecte Input
OutputCurrent
Assigned Consumed
ID Type workload Workload
Deadline faults d faults product(s) product(s) Developer(s) condition
Start condition
0 FD
128
0
24
0
0
0
1
0,1 WAIT_START 0-NOT_START
1 FDR
16
0
0
0
0
1
1
0,1,2 NOT_START
0-FINISHED
2 PG
72
0
37
0
0
1
2
0,2 NOT_START
1-FINISHED
3 PGR
9
0
0
0
0
2
2
0,1,2 NOT_START
2-FINISHED
4 MT
48
0
45
0
0
1,2
3
0
NOT_START
3-FINISHED
5 MDB
0
0
45
0
0
3
1,2
0
NOT_START
4-FINISHED
6 FT
40
0
51
0
0
2
4
0
NOT_START
5-FINISHED
7 FDB
0
0
51
0
0
4
1,2
0
NOT_START
6-FINISHED
Products’ Definition
Injected
Detected
ID Size faults
faultsCompletion
0
10
0
0
1.0
1
0
0
0
0
2
0
0
0
0
3
0
0
0
0
4
0
0
0
0
Fig. 4Developers’ Definition
ID Name
Skill WorkTime
0
Willium
10
1 Richard
1
0
2
John
1
0

Project description for the simulator

4.3 Confusion by uctuating skill level
In risky projects, the skill of a developer can uctuate(for instance, it falls down from 1.0 to 0.7). However,in the previous simulator, the value of the skill level L
was a constant determined before a simulation begins.
In the enhanced simulator, we adjust the value of L torepresent a risky situation in a project.
Let us explain an example of a risky situation ina design activity. As shown in Figure 2, transitions tcm
and tth re selectively according to the ring rates rcm
and rth . Consider the following two cases. Assume
that Kcm = 0:1, Kth = 0:2, M = 2, R = 1, and D = 1.
Assume that the assigned workload is 80 hours.
Case 1 (Ideal Case): Consider a case of no risk, inwhich the skill level L = 1. Firing rates rcm andrth are calculated as follows:rcm= 0:1 

21
= 0:2
11

1
1

r = 0:2  = 0:2
thThese two equations simulate a situation wherethat communication occurs as frequently as thethinking (rcm =rth = 0:2=0:2 = 1:0).
In our simulator, the communication among developers is treated as a verbose behavior in the activity. So, communication is modeled as a wastingof time, that is, it does not consume any workload but only takes time. The cost of an activity
increases according to the increase of time.
From the de nition of Activity model in Figure 2,
tth res 80 times during the activity, because theexecution of an activity nishes when w = 80.
Thus, tcm also res 80 times. Since the time needed
for a communication behavior is 5 minutes, it takes
400 minutes (= 6:6 hours) more than expected.
Case 2 (Risky Case): Consider a case of a risky
project where the skill level of developers becomes

L = 0:7. r

and rth are then calculated as follows:21= 0:29rcm = 0:1 
0:7  1
0:7
= 0:14
rth = 0:2 
1
In this case, communication is about 2.1 times(rcm =rth = 0:29=0:14 = 2:1) more frequent than
thinking. Thus, tcm res 168 times, and it takes
840 minutes (= 14 hours) more than expected.
As a result, since there are 2 developers(M = 2) in thisactivity, the cost of Case 2 is (14 6:6)  2 = 14:8
person-hours more than Case 1.
cm

4.4 Confusion by the deadline pressure
As mentioned earlier, in risky projects, the in uenceof deadline pressure may increase greatly. The valueof D in the previous simulator was also a constant determined when a simulation begins. In the enhanced
simulator, we adjust the value of D to represent deadline pressure.
Consider the following two cases in a design activity. Assume that Kwr = 1:0, Kin = 0:1, R = 1, F = 0,
and L = 1. Assume that assigned workload W L is 80
hours, and scheduled duration to complete the activity
is 10 days.
Case 1 (Ideal Case): Assume that 90% of workload(W L)
is consumed when 9 days elapse. Since consumption of W L is on schedule, deadline pressure does
not happen(See Case 1 in Figure 5). So, D = 1:0
holds during the activity.
In this case, ring rates for transitions twr and tin
are calculated as follows:= 1:0  1  1 = 1:0
0+1
= 0:1
rin = 0:1 
11
According to Figure 2, the number of faults injected in the current product is calculated as follows:r

wr

IEICE TRANS. FUNDAMENTALS, VOL.E82{??, NO.1 JANUARY 1999

6

Residual
WL: 8Consumed WL : 80 x 0.9 = 72

Case 19 dayswithout pressure (D=1.0)
Residual
WL: 16

Consumed WL.: 80 x 0.8 = 64

Case 29 dayswithout pressure (D=1.0)
Fig. 5

with deadline pressure (D=0.5)

Example of deadline pressure

0:1
= 7:3
0:1 + 1:0
Case 2 (Risky Case): Assume that 80% of W L isconsumed when 9 days elapse. In this case, consumption of W L is behind schedule, and thus
deadline pressure occursy (See Case 2 in Figure 5).
To represent the deadline pressure, the value of D
is decreased from 1.0 to 0.5. During the consumption of the residual 20% of W L, the ring rates arechanged as follows:f = 80 rwr= 1:0  1  0:5 = 0:5

r = 0:1 
in

0+1
= 0:2
1  0:5

The number of faults injected in the current product is calculated as follows:
0:1
f = (80  0:8) 
0:1 + 1:0
0:2
= 10:4+ (80  0:2) 
0:2 + 0:5
Faults will be removed in successive review or debug activities. If faults are detected in the review activity, it will take extra 30 minutes to remove each fault.
That is, additional 1.5 person-hours are needed to remove the faults.
Furthermore, suppose that such faults are not detected in the review, and instead are detected in thetest and debug activities. The activities are modeled
to take 1 workload to detect a fault, and 1 workload to
remove a fault. This implies that more than 6 personhours (2 workloads  3 faults) are needed to detect andremove such faults.
5.

Implementation of Enhanced Simulator

5.1 Adjusting parameters for risk factors
In order to determine how to adjust the parameters, weinterviewed the developers in the company. As a resultof the interviews, we nally determined the parameters to represent the risk factors. Table 3 shows howyThe reason why such delay occurs is out of the scopeof this example, but we can assume that a certain risk has
already existed in the activity.

the risk factors are implemented by the adjustment ofparameters.
The left side of Table 3 shows the risk factors, andthe right side shows the adjustment of parameters inthe simulator to represent corresponding risk factors.
Most adjustments are speci ed by di erences (plusor minus) with respect to the current values.
(1) For risk factors with a binary evaluation, wechanged the values of the parameters according to
one of the evaluations. For example, if the evaluation of \Requirement speci cation review" is \Notdone", the parameters for `deadline pressure' and`completion rate' are decreased by 0.1 and 0.05,
respectively, and `injected fault' is increased by 2.
(2) For risk factors with ordinal evaluation, wechanged the values of parameters according to thecorresponding order. For example, if the evaluation of \The morale of developers" is  (0   3), the skill level of developers are decreased by
0:03  (3 ).
(3) For risk factors with evaluations using an absolute value, we changed the values according to step
functions such as w1 ( ), d1 ( ) and so on. For example, if the evaluation of \Number of requirements changes" is , the workload for the projectis decreased by step function w1 ( ) where w1 ( )generates values from 0 to 3 according to .
The \review e ort ratio" is the only exception. The
value of the parameter \workload for the review activity" is set to % of the corresponding design/coding
activity according to the evaluation. The value of a parameter is cumulatively increased or decreased if several
risk factors in uence the parameter.
Let us explain an implementation of a risk factor.
If the risk factor \Requirement speci cation review"is \Not done," then the corresponding parameters inthe simulator are changed. Because of the incompleteness of the requirements speci cation, the completion
rate R is decreased by 0.05. The incompleteness of
the requirements also in uences the deadline pressure
because such incompleteness is usually revealed in thenal phase of an activity. The value of deadline pressure D is also decreased by 0.1. Since the lack of review
implies several residual faults, the number of injectedfaults F in the initial product is increased by 2yy .
5.2 Procedure of application
Here, we explain the procedure of application of theenhanced simulator to an actual project.
In order to estimate costs of projects, we prepared
inputs for the enhanced simulator. The inputs for thesimulator are a \project description" (See Figure 4) andyy

Note that the values shown in Table 3 are heuristic ones
based on our experience in the company. The generalizationof such parameter setting remains as a future work.

MIZUNO et al.: ENHANCING SOFTWARE PROJECT SIMULATOR TOWARD RISK PREDICTION

Table 3Adjustment of parameters according to evaluation result

Risk factors
Description of risk factors

Adjustments of parameters
Evaluation results
Deadline pressure(D): decrease by 0.1.
Completion rate(R) for initial product: decrease by 0.05.
Injected faults(F) in initial product: increase by 2.

Requirements specification review

Not done

Number of requirements changes# of changes (α)

Resultant products in the project plan
Project plan review
Project plan review by the manager

Not described
Not doneNot done

Requirements

Project Plan

Optimism for the technical issues

ExistsAppropriate estimation

Not done

Needed time to make estimatedays (β)

Unclear project management method
Review effort ratio

Developers

Unclear 3 - 0 Clear(δ)
Ratio in % (χ)

Communication rate (Kcm) in all activities: increase by 0.01 x δ.
Thinking rate(Kth): decrease by 0.01 x δ.
Workload for the review:
set to χ% of corresponded activity’s effort.

The morale of developers

Skill level(L) of all developers: decrease by 0.03 x (3-ε).

Experience of the requirement describer

High 3 - 0 Low(φ)

Skill level(L) of requirement describer: decrease by 0.03 x (3-φ).
Completion rate(Kcm) for initial product: decrease by 0.02 x (3-φ).
Injected faults(F) in initial product: increase by 1 if φ = 3.

Responsive person for each activity in the WBS
Untechnical pressure

High 3 - 0 Low(γ)
Not specified
Exists\evaluations of risk factors." The project description
is constructed from the plan of the target project. It
speci es the initial assignment of workload, assignmentof developers, initial parameters in the simulator, andso on.
The evaluations of risk factors are collected fromproject managers of the target projects. The parameters in the simulator are adjusted by the evaluations ofrisk factors according to Table 3. The simulator thencalculates the resultant cost for the project under therisk factors. By comparing the resulting cost in a risky
situation with an initial estimate, we can see how muchthe cost exceeds the estimate under speci ed risks.
6.

Workload for the project: decrease by w2(β) [max. 5%].
Communication rate(Kcm) in design & coding: increase by r2(β) [max. 0.05].
Deadline pressure(D): decrease by d2(β) [max. 0.1].

Strong 3- 0 Weak(ε)

Average experience of the developer

External Risks

Workload for the project: increase by w1(α) [max. 10%].
Deadline pressure(D): decrease by d1(α) [max. 0.1].
Completion rate(R) for initial product: decrease by r1(α) [max. 0.05].
Injected faults(F) in initial product: increase by f1(α) [max. 3].
Completion rate(R) for initial product: decrease by 0.05.
Completion rate(R) for initial product: decrease by 0.05.
Completion rate(R) for initial product: decrease by 0.05.
Workload for the project: decrease by 5%.
Deadline pressure(D): increase by 0.1.
Communication rate(Kcm) in the design and coding: increase by 0.05.
Thinking rate(Kth): decrease by 0.05.

Estimation

Project
Management

7

Skill level(L) of all developers: decrease by 0.03 x (3-γ).
Skill level(L) of all developers: decrease by 0.05.
Workload for the project: decrease by 10%.
Deadline pressure(D): decrease by 0.2.

Concept design Function design Structure design Module design& review& review& review& review

CD

FD

CDR

SD

FDR

MD

SDR

MT

MDB

Fig. 6

PG

MDR

PGR

IT

Moduletest & debug

Coding& review

FT

IDB
Integration
test & debug

VT

FDB

VDB

Function
Validation
test & debug test & debug

Development process of target projects

Case Study

In order to show the e ectiveness of the enhanced simulator, we made a case study. The objective of this casestudy is to show that the new simulator can estimate
the cost according to the applied risk factors.
6.1 Target projects
In the case study, we use three projects (P R1 , P R2 ,and P R3 ), each of which is a typical development ofembedded software for ticket vending machines in acertain company. They were executed in 1997. The
development process of these projects was the standard
waterfall model, as shown in Figure 6.

6.2 Estimation by previous simulator
First, we simulated the 3 projects with the previous
simulator so that the estimated costs re ect the planned
costs as correctly as possible. The estimated costs, calculated by the simulator, for the projects P R1 , P R2 ,and P R3 are shown in Table 4. For comparison, theplanned cost and the actual cost for each project arealso shown. By comparing the planned cost and estimated cost in Table 4, we can see that the simulator
can estimate the development cost quite correctly interms of the project plan for each project.

IEICE TRANS. FUNDAMENTALS, VOL.E82{??, NO.1 JANUARY 1999

8
Table 4

P R1
P R2
P R3

Estimated costs of 3 projects (person-days)

Planned cost
Estimated cost
Actual cost
Planned cost
Estimated cost
Actual cost
Planned cost
Estimated cost
Actual cost

CD
31
34
31
{
{
{
76
70
108

FD{PG
73
78
84
78
72
99
33
42
81

MT{VDB
41
50
46
14
37
19
30
31
57

Total
145
162
161
92
109
118
139
143
246

P R1
P R2
P R3

 At the beginning, a certain part of the system was
expected to be reused from other systems. However, it became clear during the functional designthat the part must be modi ed.
 An experienced developer, who knows that part ofthe system well, was not available.
 As a result, the planned delivery date of the system
was behind schedule.

6.3 Estimation by enhanced simulator
We regret to say that we do not have actual evaluations for the risk questionnaire for the 3 projects, P R1 ,P R2 , and P R3 , since these projects had already been
completed in 1997. We therefore tried to see the e ectof the implemented risk factors by constructing virtual
situations.
We constructed two cases of risks: \Ideal (no risk)
Case" and \Risky Case." The evaluations of two cases
are shown in Table 5. The evaluations for \Risky Case"are entirely bad, and for \Ideal Case" are entirely good.
The probabilities of being risky are calculated by therisk prediction model[4] for the cases \Ideal Case" and\Risky Case." They are 0.8% and 95.6%, respectively.

The problem clearly corresponds to the risk factors in
Table 2. From these facts, we consider that P R3 was a\Risky" project.
The results for the \Ideal Case" for projects P R1
and P R2 show that they are close to the actual costs.
On the other hand, estimated costs are relatively highin the case of the \Risky Case." Among them, theresults for the \Risky Case" of P R3 (that is, 207) isclose to the actual cost (that is, 246).
As a result of this case study, we validated theexecution of the enhanced simulator under risks at thecompany.
7.

Table 5Requirements
Project Plan

Estimation
Project
Management

Developers

External Risks

Two cases in the case study

Risk factors
Requirements specification review
Number of requirements changes
Resultant products in the project plan
Project plan review
Project plan review by the manager
Optimism for the technical issues
Appropriate estimation
Needed time to make estimate
Unclear project management method
Review effort ratio
The morale of developers
Experience of the requirement
Average experience of the developer
Responsive person for each activity
in the WBS
Untechnical pressure

Ideal Case
Done
0
Described
Done
Done
Not ExistsDone
50 (Clear)20
3 (Strong)
3 (High)
3 (High)

Risky Case
Not done10
Not described
Not doneNot done
ExistsNot done0
3 (Unclear)30 (Weak)0 (Low)
0 (Low)Specified

Not specified

Not exists

Exists

6.4 Discussions
Table 6 shows the estimated costs calculated by the enhanced simulator. In Table 6, \Decision by [4]" showsthe evaluation of being risky or not, based on the risky
project's decision criteria in reference [4] for the resultant project documents. Since there are no serious
problems in the documents for the projects P R1 andP R2 , they are considered as \Non-risky." On the other
hand, in the resulting documents of P R3 , the following
descriptions remain:

Comparison of development costs
Decision Actual
Estimated costby [4]
cost Ideal Case Risky Case
Non-Risky
161
149
230
Non-Risky
118
118
149
Risky
246
123
207

Table 6

Conclusion

We have proposed an enhanced project simulator thatcan evaluate development costs under risks. In the new
simulator, the parameters such as skill level of developers and the deadline pressure are adjusted according
to evaluations for the risk factors. The experimental
evaluation showed the validity of the new simulator.
The future work includes: A generalized method for determining values of parameters for risk factors should be developed.
 We must apply the new project simulator to actual

development projects, from which we can collectdata on actual risks.
 We have to investigate more risk factors that were
not included in this study.
 We must develop a simulator based method for replanning risky projects.
Acknowledgment

Authors would like to thank Associate Prof. Shinji
Kusumoto and Assistant Prof. Tatsuhiro Tsuchiya of
Osaka University for their discussions and advice to ouranalysis in this paper.

MIZUNO et al.: ENHANCING SOFTWARE PROJECT SIMULATOR TOWARD RISK PREDICTION

9

received B.E. degree in information and computer sciencesfrom Osaka University in 2001. He is currently a master course student in the Department of Informatics and Mathematical Science at Osaka University. His research interest includes the software process simulation.
Daisuke Shimoda

References[1] W. S. Humphrey, Managing the Software Process, Addison
Wesley, MA, 1989.
[2] S. Kusumoto, O. Mizuno, Y. Hirayama, T. Kikuno, Y. Takagi, and K. Sakamoto, \A new project simulator basedon generalized stochastic Petri-Net," Proc. of 19th International Conference on Software Engineering(ICSE'97),
pp.293{303, 1997.
[3] S. Kusumoto, O. Mizuno, T. Kikuno, Y. Hirayama, Y. Takagi, and K. Sakamoto, \Software project simulator for effective process improvement," Trans. of Information Processing Society of Japan, vol.42, no.3, pp.396{408, 2001.
[4] O. Mizuno, T. Kikuno, Y. Takagi, and K. Sakamoto, \Characterization of risky projects based on project managers'
evaluation," Proc. of 22nd International Conference on
Software Engineering(ICSE2000), pp.387{395, 2000.
[5] T. Furuyama, Y. Arai, and K. Iio, \Fault generation modeland mental stress e ect analysis," Journal of Systems andSoftware, vol.26, pp.31{42, 1994.
[6] F. P. Brooks, Jr., The Mythical Man-Month, Addison Wesley, MA, 1975.
[7] T. K. Abdel-Hamid, \The dynamics of software projectstaÆng: A system dynamics based simulation approach,"
IEEE Trans. Software Eng., vol.15, no.2, pp.109{119, 1989.
[8] Y. Hirayama, \Quantitative evaluation of software process
based on hierarchical project management model," Master
dissertation, Osaka University, 1996.
[9] E. Yourdon, Death March : The Complete Software Developer's Guide to Surviving `Mission Impossible' Projects,
Prentice Hall Computer Books, 1997.
[10] K. Matsumoto, S. Kusumoto, T. Kikuno, and K. Torii, \Anexperimental evaluation of team performance in program
development based on model { Extension of programmer
performance model," Journal of Information Processing,vol.15, no.3, pp.466{473, 1992.
[11] H. Sackman, W. J. Erickson, and E. E. Grant, \Exploratory
experimental studies comparing online and oine programming performance," Comm. ACM, vol.11, no.1, pp.3{11,1968.
[12] M. V. Genuchten, \Why is software late? An empirical
study of reason for delay in software development," IEEE
Trans. Software Eng., vol.17, no.8, pp.582{590, 1991.

received B.E. and
M.E. degrees in information and computer sciences from Osaka University in
1996 and 1998, respectively. He has
been working for Osaka University since1999. He is currently a Research associate in the Department of Informatics
and Mathematical Science at Osaka University. His research interests include thesoftware process and the software quality
assurance technique. He is a member ofOsamu Mizuno

the IEEE.

received M.S. and
Ph.D. degrees from Osaka University in1972 and 1975, respectively. He joined
Hiroshima University from 1975 to 1987.
Since 1990, he has been a Professor in
the Department of Informatics and Mathematical Science at Osaka University. His
research interests include the quantitativeevaluation of software development processes and the analysis and design of faulttolerant systems. He served as a program
co-chair of the 1st International Symposium on Object-Oriented
Real-Time Distributed Computing (ISORC'98) and of the 5thInternational Conference on Real-Time Computing Systems andApplications (RTCSA'98). He also served as a general co-chair of
the 2nd International Symposium on Object-Oriented Real-Time
Distributed Computing (ISORC '99).
Tohru Kikuno

received B.E. degree in information and computer science,from Nagoya Institute of Technology in1985. He has been working for OMRONCorporation.
Yasunari Takagi