Analyzing E ects of Cost Estimation Accuracy on
Quality and Productivity
Osamu Mizuno, Tohru Kikuno

Katsumi Inagaki, Yasunari Takagi,

Graduate School of Engineering Science
Osaka University
1-3 Machikaneyama
Toyonaka-City, Osaka, 560, Japan+81-6-850-6567
o-mizuno@ics.es.osaka-u.ac.jp

Keishi SakamotoSocial Systems Business GroupOMRON Corporation
Kusatsu-City, Shiga, 525, Japan+81-77-565-5071
inagaki@eftses.krc.omron.co.jp
ticket vending machines. In order to introduce a new
methodology or tool into the eld of development, itis necessary to show its e ects and usefulness, and allow developers to understand it. This study is one suchattempt.

ABSTRACTThis paper discusses the e ects of the estimation accuracy for software development cost on both the quality of the delivered codes and the productivity of thedevelopment team. The estimation accuracy is measured by metric RE (called relative error). Similarly,
the quality and productivity are measured by metrics
FQ( eld quality) and T P (team productivity). Using
actual project data on thirty-one projects at a certain
company, the followings are veri ed by correlation analysis and test of statistical hypotheses: (1) There is ahigh correlation between the faithfulness of development
plan to standards and the value of RE (A coecientof correlation between them is 00:60). (2) Both F Q
and T P are signi cantly di erent between projects with
010% < RE < +10% and projects with RE  +10%
(The level of signi cance is chosen as 0.05).

In Company A, the process improvement activity has
been conducted by the software engineering processgroup(SEPG). Especially, the following attempts havebeen carried out enthusiastically.

 Exhaustive collection of fundamental data[12][13].
 Establishing standards for activities.
{ Constructing the project plan.
{ Describing the development process[8][12].

As a result of these e orts, the following improvements
have been observed in quality and productivity.

KEYWORDS The development plan document tends to be con-software project, development plan, software quality,
productivity, test of statistical hypothesis

structed faithfully to the standard of good writing.

 The development cost, which is one of the mostimportant factors in the company, tends to be estimated accurately.
 Both the quality of the delivered code and the productivity of the development team tend to be stable
and improving.

1 INTRODUCTION

In order to achieve high quality of software products and productivity of development team, various methodologies and technologies have been proposed.
One of the most well-known methodologies is process improvement[6], and many theoretical investigations[3][11] as well as practicalapplications[7][12] have been reported. Some of them
have already been actually implemented as tools[1].

These improvements have led the development managers and the SEPG to believe the following proposition,
which we call P1 : \If the cost estimation is accurate in
the development plan, the quality of the product andthe productivity of the team become stable and high."
At present, verifying this proposition is one of the main
problems for the manager and the SEPG.

This paper describes empirical research in a certain
company, which we call Company A for convenience.
Currently Company A is mainly developing systems
with embedded software such as ATMs (AutomatedTeller Machine), POS (Point of Sales) terminals andOn the other hand, the estimation of the quality andthe productivity should be done for the project to be
executed, and so it must be done at the beginning of the
project. However, the accuracy of the cost estimation
can only be calculated at the end of the project. That is,based on the proposition P1 , it is impossible to estimate
the quality and the productivity at the beginning of
the project (even if the proposition P1 is proved to becorrect).

1

Then, we try to prove the second proposition P2 : \If
the plan is constructed faithfully to standards of goodwriting, then the cost estimation becomes accurate." In
this paper, we aim to verify these two propositions using
the data collected from actual development projects in
Company A(unfortunately, the values of collected datacannot be published in this paper because of the conditions in the contract with Company A).
In this paper, we begin by discussing the proposition P2 .
We evaluate and grade the development plan document
by applying the procedure adopted for years in SEPG.
The construction of the plan is evaluated with respect
to the faithfulness to standards of good writing(the best
grade is 50), and the development is evaluated with respect to the faithfulness to the plan(the best grade is
50).
Then, we calculate the cost estimation accuracy(called RE ) de ned by RE = (actCOST 0
estCOST )=estCOST 2 100 (%), of each project where
actCOST and estCOST are the actual cost and estimated cost, respectively. Based on the calculated valueof RE , we classify projects into three classes, C0 , C+
and C0 : C0 is a set of projects with 010% < RE <+10%, C+ is a set of projects with RE  +10% and C0
is a set of projects with RE  010%. Here, we consider
that ten percent is an important empirical threshold forthe evaluation of RE .
In order to discuss the proposition P2 , we analyze thecorrelation between the grade of the development planand the classi cation of projects based on the cost estimation accuracy RE . As a result, we show that there
is a relatively high correlation between the faithfulness of development plan to standards and the value ofRE (a coecient of correlation between them is 00:60).
The negative value of correlation coecient means thatthe higher the faithfulness to standards becomes, thesmaller the value of RE becomes(that is, the cost estimation becomes accurate).
Next, in order to discuss the proposition P1 , we analyze the following two relationships: (1) the relationship between the classi cation of projects based on RE
and the quality of the delivered code, (2) the relationship between the classi cation of projects based on RE
and the productivity of the team, by using the test of
statistical hypothesis(t-test). As the result, we showthat both the quality of the delivered code and the productivity of the team are signi cantly di erent between
projects with 010% < RE < +10% and projects withRE  +10% (The level of signi cance is chosen as 0.05).

2 BACKGROUND AND OBJECTIVE
2.1 Background

In Company A, process improvement has been undertaken to achieve high quality of software products andhigh productivity of the development team for the lastve years[12]. The main improvement activities are
summarized as follows:(IM1 ) Exhaustive but systematic collection of fundamental data from projects in Company A.
(IM2 ) Establishing several key standards for developmental activities in projects.
The improvement activity IM1 was executed according
to software metrics recommendations[4][9][13], and thefollowing D1 { D5 summarize the data to be used inthis paper.
(D1 ) the development plan document.
(D2 ) the size of delivered code(measured by Kstep).
(D3 ) the e ort of each activity of the development process(measured by person-month).
(D4 ) the number of faults detected and corrected during review, test and debug activities.
(D5 ) the number of faults in the delivered code.
Among them, data D1 is constructed before actual development starts, data D2 { D4 are taken after development is accomplished and code is delivered, and D5
is collected during the six months after the code is delivered.
Next, the improvement activity IM2 was focused on theconstruction of project plans and the formal de nition ofdevelopment process. After trial and error, Company A
has succeeded in establishing standards of good writingfor project plans. Additionally, software development
process in Company A has been formally described using the generalized stochastic petri-net(GSPN), and a
project simulator has already been implemented basedon the GSPN[8].
As the result of these improvement activities over the
past ve years, the following changes C1 { C3 have been
observed in Company A in the direction of improvementof quality and productivity.
(C1 ) The development plan document (data D1 in the
improvement activity IM1 ) tends to be constructedfaithfully to standards of good writing(establishedin the improvement activity IM2 ).

(C2) The development cost, which is one of the mostimportant factors for the company(but unfortunately is very dicult to estimate exactly[2][5]),tends to be estimated accurately.
(C3) Both the quality of the delivered codes (data D5
in the improvement activity IM1 ) and the productivity of the development team (data D2 , D3 in
IM1) tend to be stable and relatively better.
Both the development managers and the software engineering process group(SEPG) are eager to know thecauses of the changes C2 and C3 . To say more straightforwardly, they conjecture that the change C1 is thecause of the change C2 and that the change C2 is thecause of the change C3 .
2.2 Objective

In this paper, we try to show the following three assertions A1 { A3 to answer the questions from the development managers and SEPG. The assertion A1 is thesame as the proposition P2 in Section 1, and the assertions A2 and A3 correspond to the proposition P1 . For
the analysis of A1 { A3 , we apply correlation analysis and test of statistical hypothesis to actual data on
thirty-one projects at Company A.
(A1 ) In the project executed according to the projectplan constructed faithfully to standards, the estimation error of the development cost is small.
The assertion A1 implies that the faithfulness to standards in the construction of the project plan and thefaithfulness to the project plan in the development are
the main reasons for the small error in the cost estimation(that is, the di erence between the estimated cost
and the actual cost is small) of the project.
(A2 ) In the project with the accurate cost estimation,
the quality of the delivered code is high.
The assertion A2 implies that the project, for which thecost estimation error is small, delivers the nal software
product with high quality. As mentioned in subsection
2.1, we evaluate the quality of the nal software productusing the data D5 collected during the six months afterits delivery.
(A3 ) In the project with the accurate cost estimation,
the productivity of the development team is high.
The assertion A3 implies that in the project which iscompleted in accordance with estimate, the productivityof the development team is high (compared with the one
in the so-called death march projects[15]).

3 DEFINITION OF METRICS
3.1 Process Model

In Company A, many kinds of computer control systems
with embedded software are developed mainly using C
language. The typical products are ATMs(AutomatedTeller Machine) for banking applications, POS(Point
Of Sales) terminals for business applications and ticket
vending machines for railroad applications. Such products are developed under the development process
shown in Figure 1. The description of Figure 1 is oneof the outcomes of the improvement activity IM2 mentioned in subsection 2.1.
The process model shown in Figure 1 is a standard waterfall model. As will be described in detail in subsection 4.1, modi cation to the requirement speci cation isvery rare and is limited only to layout of screens or the
speed of CPU, and thus most of the requirement speci cation is decided by Company A. This may be one of themain reasons why the waterfall model shown in Figure 1
is still e ectively and successfully used in the company.
Strictly speaking, some kinds of irregular control ows(such as backwards ow to previous activity or concurrent executions between previous and current activities)
do rarely happen. But these are not explicitly describedin Figure 1.
The development process consists of two successive
phases, design phase and debug phase. One characteristic of the design phase is that the review activity
is introduced after each design activity and coding activity. The design is divided into four stages: Concept,
Function, Structure and Module. On the other hand,debug phase consists of the repetition of a pair of test
and debug activities for four di erent objectives: Unit,
Integration, Function and Veri cation.
3.2 Metrics for Software Development

In this paper, we adopt the following three kinds of metrics: SLC , F Q and TP [4][9] to analyze and evaluate
the software development from the viewpoints of thequality of software products and the productivity of theteam.
(1) Size of delivered code SLC
This metric counts the total lines of source codes including those reused, but excludes comments. Also,
the lines of reused source code are calculated according to the degree of modi cation. Thus thismetric is intended to evaluate the size of the nal
software products developed by the project.
We introduce the following six symbols:

SLC : size of delivered code

Design Phase

Debug Phase

Concept Design(CD)

Then the quality of delivered code FQ is de ned
using F D and SLC as follows:

FQ =Unit Test(UT)

Concept Design Review(CDR)

Unit Debug(UDB)

Function Design(FD)

Integration Test
(IT)

Function Design Review(FDR)

Integration Debug(IDB)

Structure Design(SD)

Function Test
(FT)

Structure Design Review(SDR)

Function Debug(FDB)

Module Design(MD)

Verification Test(VT)

Module Design Review(MDR)

Verification Debug(VDB)

Coding(PG)(3) Productivity of the team T P
This metric is intended to evaluate the average productivity of all developers in the development teamworking for the project. Therefore, it is de ned by
the ratio of the size of delivered code on the total
amount of e orts needed in the development.
We introduce the following two symbols:

T P : productivity of the team(measured by
Kstep/person-month).
EFT : the total amount of e orts needed in thedevelopment (measured by person-month).

Then the productivity of the team T P is de ned
using EFT and SLC as follows:

Coding Review
(PGR)

TP =

Figure 1: Process model

newSLC : size of code which was newly developed.

slgSLC : size of code which was modi ed slightly.
extSLC : size of code which was modi ed extremely.

The values of these symbols are measured by Kstep.

; : empirical constants.
Then the size of delivered code SLC is de ned asfollows:

SLC = newSLC +FD
SLC

2 slgSLC + 2 extSLC(2) Quality of delivered code F Q
This metric is de ned by a normalized value of thenumber of faults detected during six months after the code delivery by the size of delivered code.
Thus this metric is intended to evaluate the quality of the nal software products developed by the
project.
We introduce the following two symbols:

F Q : quality of the delivered code(measured by
the number of faults/Kstep)

F D : the number of faults detected during six
months after code delivery.

SLC
EF T

3.3 Metrics for Development Plan

(1) Cost estimation accuracy RE
Here, we use the person-month as the unit of costrather than dollar or yen. This metric is intended
to evaluate the appropriateness and feasibility ofthe development plan. Intuitively speaking, thismetric is de ned as the relative error of the costestimated at the beginning of the project from theactual cost at the end of project.
Now, we introduce the following three symbols:

RE : cost estimation accuracy(measured by %).
actCOST : the actual cost(measured by person-

month).
estCOST : the estimated cost(measured byperson-month), which is determined in development plan.
Then the cost estimation accuracy RE is de ned
as follows:

RE =actCOST 0 estCOST
2 100
estCOST(2) Faithfulness F I
This metric is intended to evaluate two kinds offaithfulness: (1) development plan was constructedfaithfully to standards of good writing (speci edby SEPG), (2) development itself was performed
faithfully to the plan (constructed by development

team). Therefore, the faithfulness F I is evaluatedfrom the two viewpoints: faithfulness to standardsof good writing staFI , and faithfulness to the plan
itself dynF I .
(2.1) Evaluation of staF I
First, the faithfulness (staF I ) of constructing the development plan is evaluated with respect to the following four components of thedevelopment plan:(a) WBS (Work Breakdown Structure) [14](b) Organization charts of project(c) PERT(Program Evaluation and Review
Technique) charts[10](d) A list of software products to be developed
Now, we explain each component in more detail. Concerning WBS, SEPG makes an inquiry into the following points: (1) level ofdescription : whether an activity in WBS isfor 2 person-months, and (2) responsibility: whether a responsible person is described
clearly for each activity. For the organization
chart, SEPG con rms the correspondence between the organization in WBS and the content of organization chart. Concerning the
PERT chart, SEPG investigates whether the
restrictions (such as development period, effort and developers) are satis ed, and whether
the critical path is described clearly. At thesame time, SEPG must con rm the correspondence between WBS and PERT chart. Finally, SEPG con rms that all of the output
product is speci ed for each activity.
(2.2) Evaluation of dynF I
Next, the faithfulness (dynFI ) of ful lling theplan is evaluated with respect to the following
two viewpoints:(e) Software review(f) Progress management
By the previous empirical study, we have conrmed that the software review has high correlation with quality of software[13], and de ned
a metric, called the ratio of review e ort(thatis, the ratio of the e orts spent in review activity to the total e orts spent in development). SEPG uses the same metric to evaluate whether review activities are executed
properly. Furthermore, SEPG con rms by interview with the developers whether progress
management has been done.
Now, we introduce the following three symbols:

F I : the faithfulness of the development plan(measured by the grade point with 0  F I 
100).

staF I : the grade point evaluated by SEPG withrespect to four components (a), (b), (c) and(d) in the plan (measured by the grade pointwith 0  staFI  50).
dynF I : the grade point evaluated by SEPG withrespect to (e) and (f) (measured by the grade
point with 0  dynF I  50).
Then, the faithfulness of the development plan isde ned as follows:

F I = staF I + dynF I
4 DATA OF DEVELOPMENT PROJECTS
4.1 Property of Projects

As described in subsections 2.1 and 3.1, the target
projects are the development of computer control systems with embedded software in Company A. The systems analyzed are classi ed into three categories: banking application, railroad application and business application.
Though we omit the details, such embedded software
implements rather complex functions dealing with many
sensors, actuators and control signals including various
kinds of interrupts. Furthermore, since it is deliveredin the form of LSI chips, modi cation of the faults after delivery is very expensive. Thus high reliability is
especially required for the embedded software.
In Company A, the development of such software is executed concurrently with the design and development ofsystem hardware. Hence it is necessary to manage thewhole project. Generally, in such a project, the speci cation of the software product is strongly in uenced by
the restrictions of the hardware design.
However, in the case of Company A, modi cation of a
speci cation will occur in some speci c and limited areas of the product, such as the layout of a screen or the
execution speed of the CPU. Fortunately, Company A
can decide the interface to the hardware and can choose
the operating system itself. As a result, the content of
the requirement speci cation of the embedded software
will be relatively stable and only changed in a very limited way. As we described in subsection 3.1, this is oneof the main reasons why the standard waterfall model
shown in Figure 1 is still e ectively used in Company A.
The targeted thirty-one projects in this study are categorized into three groups:(1) Six projects related to the banking system : ATM.

We represent and refer to the six projects by
P B1 ; P B2 ; 1 1 1 ; P B6.
(2) Twenty-four projects related to railroad system :automatic gate machine, ticket vending machine.
We represent and refer to the twenty-four projects
by PR1 ; P R2 ; 1 1 1 ; P R24 .
(3) One project related to retail system : POS terminal.
We represent and refer to it by P S1 .
The cost (that is, development e ort) of these thirty-one
projects ranges from 18.8 person-months to 185 personmonths. The average cost is 46.3 person-months.
4.2 Evaluation of Development Plan

We evaluate the metric \faithfulness" FI for the development plan. The evaluation was performed from thetwo district viewpoints: staF I (construction phase) anddynF I (execution phase). According to SEPG's judgement, ve attributes are evaluated and then the grade
points are summed up. As described in subsection 3.3,
both of staF I and dynF I range from 0 to 50.
Although there are thirty-one projects available foranalysis, only seventeen projects can be evaluated fortheir development plans. The main reason is that various kinds of defectiveness (such as missing data) occur
on some development plans. Since a few years has already passed since these projects were completed, SEPG
could not interview the actual developers of the project.
Table 1 shows the result of evaluation of development
plans. In the evaluation, we use six banking projects
PB1; 1 1 1 ; P B6, ten railroad projects P R1; 1 1 1 ; PR10
(out of twenty-four projects) and one retail project PS1 .
In Table 1, the grades A, B, C and D are introduced
to clarify cause-e ort relations between changes C1 andC2 (described in subsection 2.1). The following showscriteria for the grades:
A
B
C
D

:
:
:
:

75  F I  100
50  F I < 75
25  F I < 50
0  F I < 25

From Table 1, we can see that there are two projects
PB2, P R10 with grade A, eight projects with grade B,
seven projects with grade C. No project is with gradeD.
4.3 Evaluation of Projects

Besides the development plan described in subsection 4.2, we have collected various data according toTable 1: Evaluation of the plan
Project Name

P B1
P B2
P B3
P B4
P B5
P B6
P R1
P R2
P R3
P R4
P R5
P R6
P R7
P R8
P R9
PR10
P S1

staF I dynF I
12
39
18
39
19
23
26
26
26
19
32
12
30
23
12
25
12

35
45
40
25
35
30
30
30
30
25
25
35
20
25
25
50
35

FI Grade
47
84
58
64
54
53
56
56
56
44
57
47
50
48
37
75
47

C
A
B
B
B
B
B
B
B
C
B
C
C
C
C
A
C

metrics de ned in subsections 3.2 and 3.3. They include
the size of delivered code SLC , the number of faults
detected after delivery F D, the total e ort needed in
the development EF T , the estimated development cost
estCOST and the actual development cost actCOST .
Based on these fundamental data, we calculate the valueof key metrics, such as the quality of delivered code FQ,
the productivity of the team T P and the cost estimation
accuracy RE .
However, the values of these metrics are con dential(asalready explained in Section 1), we cannot publish thevalues themselves in this paper. Table 2 shows only theskeleton of the data with respect to these metrics.
5 ANALYSIS

In this section, we verify the assertions A1 , A2 and A3
described in subsection 2.2 using the the data in Tables
1 and 2 from the actual thirty-one projects. Before theveri cation, we classi ed the projects into three classes:
C0 , C+ and C0 , based on the value of RE . The criteriaof classi cation is borrowed from internal criteria usedin Company A. In Company A, ten percent is considered to be an important threshold for the evaluation of
RE . That is, if 010 < RE < +10, then the project issuccessful with respect to cost estimation. We classify
the projects as follows:

C0 :
C+ :
C0 :

010 < RE < 10RE  10RE  010

Table 2: Skeleton of project evaluation
Project
Name

SLC(Kstep)FD(# of faults)

RE

(%)

FQ

EFT

(# of faults/person-month)That is, the class C0 includes the projects whose cost estimation accuracy ranges from 010% to +10%, and theclasses C+ and C0 include the projects whose RE 's are
greater than +10% and less than 010%, respectively.
Table 3 shows the value of RE and the classi cationfor each project. From Table 3, the class C0 includes
seventeen projects, the class C0 includes ve projects,
and the class C+ includes nine projects, respectively.
5.1 Evaluation of Estimation Accuracy(A1 )(1) Comparison between grade and classFirst, we investigate the relationship between thethe grades A, B, C and D by the faithfulness FI
and the classes C0 , C0 and C+ by the cost estimation accuracy RE . Table 4 shows the resultant
relationship only for the seventeen projects shownin Table 1. For example, the class C0 includes two
projects with grade A, four projects with grade B,
and one project with grade C.
From Table 4, we see that all projects with grade A
belong to the class C0 . This provides limited evidence that if the development plan was constructedfaithfully to standards and the development was
performed faithfully to the plan, then the cost estimation of the corresponding project is very accurate.
On the other hand, we can observe that the most of
projects with grade C tend to belong to either theclass C0 or the class C+ . This implies that if thedevelopment plan was not constructed faithfully tostandards or the development was not performed
faithfully to the plan, then the cost estimation is
not accurate, neither.
(2) Further analysis of FI and RE
Next, we investigate to a greater extent the relationship between F I and RE . In this further analysis, we take projects from the same application, andconsider ten projects P R1 ; P R2 ; 1 1 1 ; P R10 (whichappear both in Table 1 and Table 3).
Then, we calculate the correlation coecient between the F I and RE for the selected ten projects

estCOST

actCOST(person-month) (person-month) (person-month)

TP(Kstep/person-month)

P R1 ; P R2 ; 1 1 1 ; PR10 . The calculated values areshown as follows:

the correlation coecient between F I and RE= 00:47
the correlation coecient between staF I and RE= 00:60

The rst result(00:47) implies that there are someextent of correlation between the faithfulness of development plan F I and the cost estimation accuracy RE (though it is limited to the railroad related
projects). Furthermore, the second result(00:60)
implies that there are relatively high correlation between the faithfulness to standards of good writingof the development plan staF I and the cost estimation accuracy RE . As a result, we can conclude
the assertion A1 is proved armatively.
5.2 Quality of Delivered Code F Q(A2 )Here, we analyze the assertion A2 about the quality ofdelivered code using the test of statistical hypothesis.
In the analysis we apply all data from the thirty-one
projects shown in Table 3.
Now, for the test of statistical hypothesis, we de ne 0
to be the average of F Q's of all projects which belong
to the class C0 . Similarly, we de ne + and 0 to be
the averages of F Q's of all projects in the classes C+
and C0 , respectively.
(1) Classes C0 and C+
We de ne two hypotheses H0 and H1 for two classes
C0 and C+. The level of signi cance is chosen as
0.05.
Null hypothesis H0 : + = 0
Alternative hypothesis H1 : + > 0
The test statistic is calculated byx 0 x
T (x0 ; x+; s20 ; s2+) = r + 0 = 1:997
s2+ s20
N+ + N0

Table 3: Evaluation of cost estimation accuracy
Estimation accuracy
Project Name RE (%)
Class
P B1
-21
C0
P B2
6
C0
P B3
19
C+
P B4
-25
C0
P B5
3
C0
P B6
-17
C0
P R1
1
C0
P R2
-2
C0
P R3
-5
C0
P R4
-17
C0
P R5
14
C+
P R6
24
C+
P R7
1
C0
P R8
32
C+
P R9
21
C+
P R10
9
C0
P R11
26
C+
P R12
2
C0
P R13
-1
C0
P R14
0
C0
P R15
13
C+
P R16
9
C0
P R17
-29
C0
P R18
6
C0
P R19
6
C0
P R20
-1
C0
P R21
-3
C0
P R22
11
C+
P R23
-2
C0
P R24
0
C0
P S1
11
C+
where x0 , s0 and N0 are the average of F Q, thestandard deviation of FQ and the number of sample projects in the class C0 , respectively. Similarly,x+, s+ and N+ are the average, the standard deviation and the number of sample projects in theclass C+ , respectively. (As we have explained, thecontract with Company A prohibits us from mentioning the values x0 , x+ , s0 and s+ calculated forthe test.) Then the distribution of T (x0 ; x+ ; s20 ; s2+ )is the t-distribution with the degree of freedom

m=

s2

2

( N++ + Ns00 )2= 14:6 < 15:s4+s40
N+2 (N+ 01) + N02 (N0 01)

From the t-distribution with the level of signi cance = 0:05, the critical region becomes R =t15 (0:05) = 1:753. Then,

T (x0; x+; s20 ; s2+) > R

Table 4: Relationship between F I and RE
Classi cation by RE
Grade
by FI

A
B
C
D

C0 C0
0
2
2
0

2
4
1
0

C+
0
2
4
0

holds. Thus, the null hypothesis H0 is rejected.
This result implies that there is a signi cant di erence on the quality of delivered code, F Q, between
the projects in the class C0 (cost estimation accuracy ranges from 010% to +10%) and the projects
in the class C+ (cost estimation accuracy is greaterthan +10%). Thus, for projects in the classes C0
and C+ , the assertion A2 is proved armatively.
(2) Classes C0 and C0
However, from the test of statistical hypothesis, it
is shown that there is no signi cant di erence on
the quality of delivered code between the projects
in the classes C0 and C0 .
5.3 Productivity of Team T P (A3 )Finally, we analyze the assertion A3 about the productivity of the team using the test of statistical hypothesis. In this analysis we also apply all data from thethirty-one projects. For the test, we de ne 0 to be
the average of T P 's of all projects which belong to theclass C0 . Similarly, we de ne + and 0 to be the averages of T P 's of all projects in the classes C+ and C0 ,respectively.
(1) Classes C0 and C+
We de ne two hypotheses H0 and H1 for two classes
C0 and C+ . The level of signi cance is chosen as
0.05.
Null hypothesis H0 : + = 0
Alternative hypothesis H1 : + < 0
The test statistic is calculated byx 0 x
T (x0 ; x+; s20 ; s2+) = +2 0 = 2:743
s+ s20N+ + N0r

where x0 , s0 and N0 are the average of T P , thestandard deviation of T P and the number of sample projects in the class C0 , respectively. Similarly,x+ , s+ and N+ are the average, the standard deviation and the number of sample projects in theclass C+ , respectively. (In this case also, we cannot show the values x0 , x+ , s0 and s+ calculated in
the proof by the same reason.) Then the distribution of T (x0 ; x+ ; s20 ; s2+ ) is the t-distribution withthe degree of freedom m = 14:6 < 15. From the tdistribution with the level of signi cance = 0:05,
the critical region becomes R = t15 (0:05) = 1:753.
Then,
T (x0; x+; s20 ; s2+) > R
holds. Thus, the null hypothesis H0 is rejected.
This result implies that there is signi cant di erence on the productivity of the team between the
projects in the class C0 (cost estimation accuracy
ranges from 010% to +10%) and the projects in theclass C+ (cost estimation accuracy is greater than+10%). Thus, for projects in the classes C0 and
C+, the assertion A3 is proved armatively.
(2) Classes C0 and C0
However, from the test of statistical hypothesis, it
is shown that there is no signi cant di erence on
the productivity of team between the projects inthe classes C0 and C0 .
6 CONCLUSION

In this paper, we have proved three interesting assertions A1 , A2 and A3 as the results of empirical research. Although the implications by these assertions
themselves may not be new for academia people, they
may become a driving force in the software developing
company for promoting process improvement through(1) exhaustive collection of fundamental data, and (2)establishing some kinds of standards (mentioned in Section 1).
The main result of our empirical research are summarized as follows: If the development plan in a project is constructedcarefully and faithfully to standards of good writing, then it is associated with more accurate costestimates.
 If the cost estimates of a project are accurate, thenthe quality of the delivered code in the project ishigher and the productivity of the team is also
higher.
The future research work includes the following: Development of procedures (or algorithms) for evaluating the faithfulness dynF I objectively.
 Detailed analysis about the projects in the classes
C+ and C0 to detect human factors.
 Investigation of method to feedback the analysis
result to actual development.
ACKNOWLEDGEMENTS

Authors would like to thank Prof. Koji Torii of Nara
Institute of Science and Technology and Dr. Shinji
Kusumoto of Osaka University for their discussions andadvises to our analysis in this paper.

REFERENCES[1] Aoyama M. : \Agile software process model",Proc. of the 21th Annual International Computer
Software and Applications Conference, pp.454{
459 (1997).
[2] Basili V. R., Condon S. E., Emam K. E., Hendrick R. B. and Melo W. : \Characterizing
and modeling the cost of rework in a library of
reusable software components", Proc. of the 19thInternational Conference on Software Engineering(the 19th ICSE), pp.283{291 (1997).
[3] Cimitile A. and Visaggio G. : \A formalism forstructured planning of a software project", International Journal of Software Engineering andKnowledge Engineering, Vol.4, No.2, pp.277{300(1994).
[4] Fenton N. E. and P eeger S. L. : \Software Metrics : A Rigorous & Practical Approach", PWS
Publishing (1997).
[5] Heemstra F. J. : \Software cost estimation",
Information and Software Technology, Vol.34,
No.10, pp.627{639 (1992).
[6] Humphrey W. S. : Managing the Software Process, Addison Wesley, Reading, MA (1989).
[7] Humphrey W. S., Snyder T. and Willis R. : \Software process improvement at Hughes Aircraft",IEEE Software, Vol.8, No.4, pp.11{23 (1991).
[8] Kusumoto S., Mizuno O., Hirayama Y., Kikuno
T., Takagi Y. and Sakamoto K.:\A new software
project simulator based on generalized stochastic
petri-net", Proc. of the 19th International Conference on Software Engineering(the 19th ICSE),
pp.293{302 (1997).
[9] Moller K. H. and Paulish D. J. : \Software Metrics : A Practitioner's Guide to Improved Product Development", IEEE Press (Chapman & Hall
Computing) (1993).
[10] Poiaga L. : \Operations research in project management and cost engineering : An outlook for
new operational developments", European J. Operational Research, Vol.41, No.1, pp.1{14 (1989).
[11] Podorozhny R. M. and Osterweil L. J. : \Thecriticality of modeling formalisms in software design method comparison", Proc. of the 19th International Conference on Software Engineering(the19th ICSE), pp.303{313 (1997).

[12] Tanaka T., Sakamoto K., Kusumoto S. andKikuno T. : \Improvement of software process
by process visualization and bene t estimation",
Proc. of the 17th International Conference on
Software Engineering(the 17th ICSE), pp.123{
132 (1995).
[13] Takagi Y., Tanaka T., Niihara N., SakamotoK., Kusumoto S. and Kikuno T. : \Analysisof review's e ectiveness based on software metrics", Proc. of the 6th International Symposium
on Software Reliability Engineering, pp.34{39(1995).
[14] Tausworthe R. C. : \The Work Breakdown Structure in software project management", Journal of
Systems and Software, Vol.1, pp.181{186 (1980).

Death March : The Complete
Software Developer's Guide to Surviving `Mission Impossible' Projects, Prentice Hall Com-[15] Yourdon E. :puter Books (1997).