First International Symposium on Empirical Software Engineering and Measurement

Is This Cost Estimate Reliable? – The Relationship between
Homogeneity of Analogues and Estimation Reliability
Akito Monden1,
Nahomi Kikuchi2,
Michael D. Barker1
Naoki Ohsugi1,
Masateru Tsunoda1,
Takeshi Kakimoto1
andKen-ichi Matsumoto1
1
Graduate School of Information Science, Nara Institute of Science and Technology
2
Software Engineering Center, Information-technology Promotion Agency, Japan{naoki-o, akito-m, mbarker, masate-t, takesi-k, matumoto} @is.naist.jp
n-kiku@ipa.go.jp
between individual projects. For example, Toda et al.
[23] estimated the cost of testing for 68 projects in the
ISBSG data repository [13]. In their case, a stepwise
regression model accurately estimated 30 projects with
30% or less relative errors, while estimates for 13 otherprojects had more than 100% errors. This suggests
project managers must face difficulty in trusting theindividual estimate even though the average estimation
error is low. If they knew the estimation reliability or
uncertainty for an individual project, it would
accelerate the diffusion of formal estimation methods.
In this paper, we experimentally test a hypothesis
about the reliability of analogy-based cost estimation“more homogenous analogues for a project produce amore reliable cost estimate.” This hypothesis is basedon the fact that analogy-based methods typically use an
average (or a weighted-sum) of the analogues’ costs toderive the target’s cost estimate. We hypothesize thatthe estimated cost is unreliable when the analogues’
costs vary widely. For instance, consider the following
two cases: the analogues’ costs in person-hours are{2900, 3000, 3100}; or they are {500, 2000, 6500}.
Although in both cases the estimated cost is 3000 (byaveraging analogues’ costs), the latter seems to be lessreliable. In this paper, the homogeneity of analogues isevaluated with a statistical measure, Coefficient of
Variation of the analogues’ costs. Then, therelationship between these measures and the estimation
reliability is investigated experimentally.
To test the hypothesis, we implemented an analogybased cost estimation method by combining advanced
features of the typical analogy-based tools ESTOR [15],ANGEL [18] and ACE [26]. At first, the implemented
method searches a project data repository foranalogues by the same manner of ANGEL; i.e.
Euclidean distance between completed projects and thetarget is calculated. Once the analogues are determined,

Abstract
Analogy-based cost estimation provides a usefuland intuitive means to support decision making insoftware project management. It derives a costestimate required for completing a project frominformation about similar past projects, namely theanalogues. While on average this method provides a
relatively accurate cost estimate there remains apossibility of large estimation errors. In this paper, we
empirically tested the hypothesis that “using morehomogeneous analogues produces a more reliable costestimate” using a software engineering data repository
established by the Software Engineering Center (SEC),
Information-technology Promotion Agency, Japan.
This testing showed that low and high homogeneity
projects had a large variation in estimation reliability.
For instance, the difference was 22.9% (p = 0.021) interms of percentage to get accurate estimates (better
than Median of Magnitude of Relative Error).

1.

Introduction

Formal cost and effort estimation methods havebeen proposed both as the basis for project schedulingand the baseline for bidding on external contracts.
Formal estimation methods derive a cost estimate froma project’s observable characteristics or projectfeatures [3]. Typical examples include function points,
work period and engineers’ skill levels. These projectfeatures are measured together with the actual costrequired to complete a project; then, this information isstored in a project data repository for use in deriving
new cost estimates.
On average, these methods provide an accurate costestimate; however the accuracy varies enormously

0-7695-2886-4/07 $20.00 © 2007 IEEE
DOI 10.1109/ESEM.2007.31

384

a linear size extrapolation is performed by the samemanner of ACE; i.e. individual analogues’ costs areadjusted along a project size feature such as function
points. Then, the cost estimate is calculated byaveraging analogues’ adjusted costs. The Euclidean
distance offers the benefits of clarity and consistency
over human judgement alone. The linear size
adjustment attempts to account for the influence on
cost of the difference in size between the target and thecompleted projects [26]. The utilized method combinesthese advantages to make accurate estimations.
We applied this method to a project repository
called the SEC data repository [19] established by the
Software Engineering Center (SEC), Informationtechnology Promotion Agency, Japan. As of March
2005, the SEC data repository consists of 1009
projects from 15 Japanese software companies. We
selected 14 project features available at the time ofestimation (at the end of architecture design phase) andthat contained less missing values than the others.
Then, we extracted for estimation 104 projects having
all of the selected features and the completion cost(effort).
The remainder of this paper is structured as follows.
Section 2 introduces the background of the study,
especially about estimation reliability and analogybased cost estimation. Section 3 explains the
hypothesis to be analysed more precisely. Section 4
explains an analysis for the hypothesis testing,especially regarding the used data and the experimental
procedure. Then, Section 5 presents the results; andSection 6 discusses the implications given by the
results. Section 7 concludes the paper and provides
directions for further research.

2.

where ca is the cost actually spent to complete a projectpa, and ea is its cost estimate. MRE is usuallycalculated for many projects in a data repository undera standard evaluation process such as cross-validation[5]. Then, its mean and median values are used as
evaluation criteria; Mean MRE (MMRE) and Medianof MRE (MdMRE).
Estimations reliability is usually assessed by
exploring reasons or risk factors for estimation error,
and then checking whether the target project has thepotential risks. Risk factors have been investigated byquestionnaire-based approaches [17], [25], [20] and bystatistical analysis [11], [7], [10]. For example, Ledererand Prasad [12] reported that one of the mostimportant risks is having “overlooked tasks” inmeasurement. In this approach, if the possibility of
having the risk is high, the estimation reliability is
considered low. One of the difficulties with thismethod is that the assessment of some kinds of risks is
not easy in principle. For example, the possibility of
having “overlooked tasks” in past measurement isoften not clear at the time of making an estimate. In
comparison with these previous studies, our study
focuses on an easy-to-assess aspect of a project, thehomogeneity of analogues, to avoid the difficulty ofassessment.
There also have been several attempts to build
formal methods to assess the estimation reliability
systematically [2], [9]. These studies proposed
methods to demonstrate the estimation reliability witha confidence interval of the cost estimation. A
confidence interval consists of a maximum and aminimum bound of cost estimate and its confidence
level. Jørgensen and Sjøberg [9] proposed a method to
identify the confidence interval using the distributionof previous estimation accuracy. Angelis and Stamelos[2] developed a tool identifying the confidence interval
by repeating accuracy evaluation with bootstrap
sampling. Our study shares their goal to provide asystematic and objective way to assess the estimation
reliability for an individual project, but the approach isdifferent. Our contribution is to provide practitioners
with clear and intelligible evidence of estimation
reliability so that they become more confident intrusting/distrusting the derived cost estimate. We
believe it is useful to develop several different types ofreliability assessment methods to promote the use offormal estimation methods in the field.

Background2.1. Estimation Reliability
In this paper, we define estimation reliability as theprobability that the estimation error becomes smaller
than a threshold value. The threshold can be given bysome standard evaluation criteria such as Mean
Magnitude of Relative Error (MMRE) or Median ofMagnitude of Relative Error (MdMRE). As the basis ofthese criteria, the Magnitude of Relative Error (MRE)
is defined as the following equation (1) indicating thedegree of estimation error against the actual cost:c − eaMRE = aca

2.2. Analogy-based Cost Estimation(1)

We focus on analogy-based cost estimation, which
is an application of Case-Based Reasoning (CBR) in
software engineering [26]. CBR emerged as a major385

research area within artificial intelligence [6], and was
first introduced to software development costestimation by Mukhopadhyay et al [15]. The processof analogy-based cost estimation is as follows. First, as
inputs to estimation, observable project features of atarget (ongoing) project are selected. Then, one ormore analogues having similar features to the target
project are found in a project data repository. Finally, a
cost estimate is derived using the known costs of theanalogues. Our expectation is that the analogues
contain valuable information about the reliability of
estimation for the target project.
We have developed a prototype tool Trinity toanalyze the nature of analogy-based estimation. Trinity
makes estimates based on techniques used in three
typical tools ESTOR [15], ANGEL [18] and ACE [26]as follows. At first, Trinity calculates the Euclidean
distance between the target project and the completed
projects, and chooses the closest k-projects as theanalogues [15][18]. Here, the number k is selected by
the same method as used in ANGEL, an exploratory
search to minimize MMRE for a particular project
repository. Once the analogues are determined, Trinity
then adjusts each analogue’s cost according to function
points. Finally, Trinity derives the cost estimate byaveraging the adjusted costs of the analogues.
More precisely, if each project pi is described by aset of values {ci, vi, 1, vi, 2, ..., vi, m}, Trinity calculates
the Euclidean distance dist(pa, pi) between the target
project pa and each completed project pi, as:dist ( pa , pi ) =

∑ (v
j∈F

a, j

− vi , j )3.

The hypothesis to be tested is “the higher the
homogeneity of the analogues, the more reliable a
project’s estimate is.” In this paper, we attempt tomeasure the homogeneity by a basic statistical metric,
Coefficient of Variation (CV) of analogues’ costs, i.e.,the homogeneity is calculated as:CV =1 ⎛ fpa ⎞∑ ⎜ ci ⎟k i∈ A ⎜⎝ fpi ⎟⎠

δA
1
=
µA µA

1
∑ (ci − µ A )2
k − 1 i∈A

(4)where A denotes a set of analogues, δ A and µ A denotethe standard deviation and the arithmetic mean of theanalogues’ costs, k denotes the number of theanalogues, c i denotes each analogue’s cost. CV is ameasure of dispersion of a probability distribution,
defined as the ratio of the standard deviation to themean. It expresses how spread out the values are [16].
In addition, it allows comparison of the variation of
populations that have significantly different meanvalues because it is a dimensionless number. Note that
CV has converse relation to homogeneity. In otherwords, a smaller CV indicates higher homogeneity.
We consider there are two typical cases where
analogues’ costs become heterogeneous. The first caseis that the analogues have some measured projectfeature causing cost uncertainty; and the second case isthat the analogues have potential uncertainty in projectfeatures that are unavailable at the time of estimation.
We explain these cases with three toy examples(Figure 1 to 3). Each figure assumed that thecompletion cost and three project features (FP, FP
Counting Method and Architecture Type) wereavailable. Also, each figure assumed that three projects
were found as analogues having feature values that arevery similar to each target project. Although theestimated costs were 3000 in all cases, only in the
Figure 1, CV of the analogues’ costs was small 0.03
(i.e. homogeneous).
Figure 1 shows a case where homogeneous
analogues were found. These contribute to make highreliability estimation if our hypothesis is true.
Figure 2 shows a case where project features cause
cost uncertainty. In this example, we assume the FP
counting method “NESMA” is less reliable than
“IFPUG” since NESMA is usually used in a more
upstream development phase. And thus the accuracy of
FP is low in this case. It can often happen that themeasured FP differs from actual FP [8]. In this case,the project feature “NESMA” caused cost uncertainty,
resulting in heterogeneity of analogues’ costs. These
analogues are considered to cause a low reliable
estimation.

(2)

where F denotes a set of project features observed at
the time of estimation, and vi, j denotes the project pi’s
j-th feature value. Then, Trinity derives the target
project pa’s cost estimate ea, as:ea =Hypothesis(3)

where A denotes a set of the analogues, k denotes thenumber of the analogues; ci and fpi denote each similar
project pi’s cost and its function points. Before thecalculation, all categorical features are transformed asdummy variables. In addition, all ranges of values arenormalized to interval [0, 1] to equalize the influenceof each project feature for similarity computation [3].
Note that this paper does not propose any analogybased estimation method. Trinity is true to its name,consisting of the great contributions of three
representative studies in the area.

386

FP Counting Architecture
MethodType

FP Counting Architecture
MethodType

CV = 0.88

Cost

FP

Stand-alone

Project 4

1000

198

NESMA

Stand-alone

IFPUG

Stand-alone

Project 5

2000

200

NESMA

Stand-alone

IFPUG

Stand-alone

Project 6

6000

202

NESMA

Stand-alone

CV = 0.03

Cost

FP

Project 1

2900

198

IFPUG

Project 2

3000

200

Project 3

3100

202

Figure 1. The analogues are homogeneous

Figure 2. The analogues have projectfeatures causing cost uncertainty

FP Counting Architecture Overlooked
Tasks
MethodType

CV = 0.88

Cost

FP

Project 7

1000

198

IFPUG

Stand-alone

0

Project 8

2000

200

IFPUG

Stand-alone

500

Project 9

6000

202

IFPUG

Stand-alone

4000

Figure 3. The analogues have potential uncertainty in unavailable project features

4.2. Analyzed Data

Figure 3 shows another case where heterogeneous
analogues were found, but in this case the cause ofheterogeneity was in the unavailable project features.
In this example, both project 8 and 9 spent cost forsome special purpose, namely Overlooked Tasks,
which was not measured. This resulted in theheterogeneity of the analogues’ costs. These analogues
are also considered to cause a low reliable estimation.

4.

The source of data used in the analysis was the2005 version of the SEC data repository containing
1009 projects from 15 Japanese software companies.
Projects completion dates ranged from 1996 to 2005.
All of the projects are custom enterprise softwaredevelopments. Each project is characterized by about
400 features, although many include a large number ofmissing values. The business domains of these projects
include manufacturing, communications, wholesale
and retail sales, finance and insurance, etc. The variety
of projects in the data covers a large part of theJapanese enterprise software industry. We believe that
the SEC data repository is a good test bed since it ismaintained by measurement specialists who keep on
improving the data quality based on follow-up surveys
and interviews with companies who collected data.
This data has been used previously in some studies
such as a performance comparison of estimation
methods [22] and the productivity analysis [24].
We selected 14 project features available at the timeof estimation and that contained less missing values
than the others. Then, we extracted for estimation 104
projects having all of the selected features and thecompletion cost (effort). We assumed that theestimation is done at the end of the architecture designphase, which is the next phase of requirement analysis.
So, first, the unavailable project features at the time ofAnalysis for Testing Hypothesis

4.1. Overview
We experimentally tested the hypothesis describedin Section 3. In the analysis, Trinity was used toestimate the total cost of each project in the SEC datarepository. The accuracy and the reliability of eachestimated cost were evaluated by leave-one-out crossvalidation [1]. Afterwards, each estimated cost wasclassified into three levels (Low / Medium / High)according to the homogeneity of analogues used forthe estimation. Then, the differences of the accuracy
and the reliability between the levels were tested withstatistical test methods such as Welch’s t-test, MannWhitney U test and Fisher’s exact test [14]. At last, thedispersion of the estimation accuracy was visuallyshown with the Outlier box plot [21] to confirm thatthe results of the statistical test meet the analyzers’
subjective evaluation.

387

Table 1. Project Features in the Analyzed Data
Name of Feature

Type

Unit / Range / Values

Cost
FP
FP Counting Method
Objective
Product Type

Numerical
Numerical
Categorical
Categorical
Categorical

Customer’s Business

Categorical

Application Domain
Number of Users
Commercial Package Use
Main Architecture Type
Main Platform# of Platforms
Main Development Language# of Languages
Lifecycle Model

Categorical
Categorical
Categorical
Categorical
Categorical
Numerical
Categorical
Numerical
Categorical

In Person Hours
Function Points
IFPUG Unadjusted, NESMA Estimated
New Development, MaintenancePackage Software, Company Specific System
Manufacturing, Communications, Wholesale and Retail, Financeand Insurance, etc.
Accounting, Marketing, Customer Management, Analysis, etc.
Limited, Unlimited
Yes, No
Stan-alone, Mainframe, 2Tier Client-Server, Internet, etc.
Windows, Unix, Other[1, 3]COBOL, C, Visual C++, C#, Visual Basic, Java, etc.
[1, 3]
Waterfall, Iterative, Others

estimation were removed from the data repository.
Another point of this extraction is to avoid influencesof the missing values on the analysis results.
Table 1 shows the selected project features. The
column labelled Type contains the scale level of eachfeature as either “Numerical” or “Categorical”. The
column labelled Unit / Range / Values contains eachfeature’s concrete values eligible for disclosure to the
public. The Cost is the total cost required forcompleting a project, and is the estimation target inthis analysis. The FP is the function point countexpressing the size of the main product developed in
the project. The Customer’s Business is the main
business of the customer in each project. The
Application Domain is the situation or the purpose of
the use of the product. In the Number of Users, thevalue of “Unlimited” means the product is for generalpublic while “Limited” means some specific customers.
The # of Platforms and # of Languages become two or
more if the product was developed with a combinationof some platforms and languages.

Step 1. The cost of each project in the dataset wasestimated with the Trinity (leave-one-out
cross validation [1]).
Step 2. The accuracy of each estimate was evaluatedas MRE via the equation (1).
Step 3. MMRE and MdMRE of all projects are usedas the threshold values, which separate thereliable estimate from unreliable estimate.
Step 4. The homogeneity of analogues for each costestimate was evaluated in terms of CV via theequation (4). Instead of each analogue’s raw
cost in the equation (4), the adjusted cost
obtained from the equation (3) was used.
Step 5. The cost estimates were categorized into threelevels (Low / Medium / High) according to
the calculated CV of the analogues’ costs.
Note that larger CV level means lower
homogeneity of analogues.
Step 6. For each level, each Level’s Mean MRE(LMMRE) and each Level’s Median of MRE(LMdMRE) were calculated as the averageand the median of MREs of the projects
included in each level. For instance, LMMRE
was calculated as:

4.3. Analysis Procedure
The hypothesis was tested by the systematic
procedure explained in this section. There was lesspossibility that the analysis results were affected by the
analyzers’ decisions, because most steps in theprocedure were automatically executed with Trinity.
The analysis consisted of nine steps as described
below:∑ MRE

LMMRE = i∈P

n

i

(5)

where P denotes a set of projects included in agiven level, n denotes the number of theseprojects, and MREi denote the MRE of thecost estimate for a project pi.

388

Step 7. For

each level, Reliability of MMRE
(RMMRE) and Reliability of MdMRE(RMdMRE) were calculated as:∑ ri⎧ r = 1, MREi < MMRE
RMMRE = i∈P × 100(% ) ⎨ in⎩ri = 0, MREi ≥ MMRE

(6)

∑ ri
⎧ r = 1, MREi < MdMRE
RMdMRE = i∈P × 100(% ) ⎨ in⎩ri = 0, MREi ≥ MdMRE(7)

MRE, so that lower points indicate lower estimation
errors, i.e. higher accuracy. Each box stretches fromthe 25th percentile to the 75th percentile of MRE,
therefore it contains the middle half in the distributionof each level. The LMMRE is shown as a plus (+)
mark on the vertical line. The LMdMRE is shown as aline across the box. A vertical line is drawn in thecenter of the box from the smallest MRE to the largest
MRE excluding the outliers. The outliers having 1.5
times the difference between the 25th and the 75th
percentile are drawn as small circles. In this case, lowhomogeneity (high CV level) has three extremely large
outliers (5.0 or more MREs) outside of Figure 4. The
numbers at the right side of the box indicate the valueof the representative points including the LMMRE, theLMdMRE, the 25th and the 75th percentile, the top
and the bottom of the vertical lines.

where the definitions of P, n and MREi are thesame as Step 6.
Step 8. The differences of LMMRE, RMMRE,
LMdMRE and RMdMRE between each twolevels were statistically tested. The statistical
test methods were appropriately chosen
according to the type of the tested values.
LMMRE was tested with Welch’s t-test,
LMdMRE was tested with Mann-Whitney U
test, and RMMRE and RMdMRE were tested
with Fisher’s exact test.
Step 9. The dispersion of MRE in each level wasgraphed with a outliner box plot to confirm
the results of statistical tests.

5.

6.

Discussion

For both LMMRE and LMdMRE, when theanalogues’ homogeneity was low, the estimation
reliability was relatively lower than the others; andthere were statistically significant differences (p < 0.1).
For instance, in terms of RMMRE, the difference
between medium and low homogeneity was 19.4% (p= 0.032); and the difference between low and high was
17.1% (p = 0.051). This tendency was consistent with
RMdMRE, the difference between medium and low
homogeneity was 15.8% (p = 0.052); and thedifference between low and high was 22.9% (p =0.021). In addition, Figure 4 visually shows clear
differences of estimation accuracy between
homogeneity low and the other levels. We believethese results strongly support our hypothesis.
On the other hand, statistical significance was notobserved in the difference between high and medium
homogeneity. In terms of MdMRE, the estimation
reliability was higher when the homogeneity was high;however the relationship was converse in MMRE.
Also the observed p-values show there was nodifference in them. These results suggest theestimation reliability became extremely lower when
the homogeneity was lower than the certain degree. So,
there would be an “optimized” threshold for CV todistinguish the unreliable estimates from reliable ones.
Actually, if a threshold was put between the low andthe medium homogeneity, i.e., between CV level ofhigh and medium, large differences can be observed in
the estimation reliability. For instance, the differenceof RMMRE is 18.25% (nearly equal to the averageof 19.4% and 17.1%); and the difference of RMdMRE
is 19.35% (nearly equal to the average of 15.8% and22.9%).

Results

Table 2 shows the relationship between estimation
reliability and homogeneity. Each column denotes each
degree of homogeneity (i.e. CV level), “Range of CV”
row contains the minimum and the maximum value of
CV in each level, “# of Projects” row contains thenumber of projects in each level. The other rows
contain the values of LMMRE, RMMRE, LMdMRE
and RMdMRE.
Table 3 shows the difference of estimation
reliability between degrees of homogeneity. Eachcolumn denotes the difference between each twodegrees of homogeneity. Each cell contains thedifference of LMMRE, RMMRE, LMdMRE andRMdMRE between corresponding column’s degrees.
Each cell also contains a p-value in parentheses for thedifference. For example, the rightmost bottom cell
contains difference of RMdMRE between Low andHigh homogeneity, and p-value of the difference. If thedifference was statistically significant at the 10% level(p < 0.1), the p-value is underlined. In addition, if avalue is quoted in the body text, it is emphasized withbold font.
Figure 4 shows MRE dispersion with respect tohomogeneity of analogues as an outlier box plot
diagram. The horizontal axis denotes the degree ofhomogeneity (CV level). The vertical axis denotes389

Table 2. Relationship between Estimation Reliability and Homogeneity of Analogues
Homogeneity

High

MediumLow

CV Level

Low

MediumHigh

Range of CV[0.000, 0.705)[0.705, 0.944)[0.944, 3.225]# of Projects

35

34

35

LMMRE

0.747

0.607

1.515

RMMRE

77.1%

79.4%

60.0%

LMdMRE

0.364

0.499

0.770

RMdMRE

60.0%

52.9%

37.1%

Table 3. Difference of Estimation Reliability between Degrees of Homogeneity
Homogeneity

High vs Medium

Medium vs Low

Low vs HighCV LevelLow vs Medium

Medium vs HighHigh vs Low

LMMRE

0.140

0.908

0.768

(p-value)( 0.418 )( 0.041 )( 0.092 )

RMMRE

2.3%

19.4%

17.1%

(p-value)( 0.294 )( 0.032 )( 0.051 )LMdMRE

0.135

0.270

0.406

(p-value)( 0.933 )( 0.023 )( 0.034 )RMdMRE

7.1%

15.8%

22.9%

(p-value)( 0.137 )( 0.052 )( 0.021 )** Each ( ) contain p-value. The underlined p-values indicate the difference was statistically significant
at the 10% level (p < 0.1). The values emphasized with bold font are quoted in the body text.

MRE 2.5

2.475

2.0
1.768
1.515
1.361

1.5
1.278
1.0

0.957

0.5
0.364

0.483

0.242

0.165
0.0

0.770

0.753
0.607
0.499

0.747

0.007

0.135

0.034

High

MediumLow

Homogeneity

Low

MediumHigh

CV Level

Figure 4. Outlier Box Plot of MRE Dispersion with Respect to Homogeneity of Analogues

390

7.

Conclusion

1974.

In this paper we have experimentally tested ahypothesis “using more homogeneous analogues for a
project to be estimated produces a more reliable costestimate.” The homogeneity is evaluated withCoefficient of Variation (CV) of the analogues’ costs.
The hypothesis was statistically tested based on costestimates obtained by applying the analogy-based
approach to the dataset containing 104 projects eachhaving 14 project features.
As a result, a large variation in reliability wasobserved between high and low homogeneity levelprojects. For instance, in terms of Reliability of Mean
Magnitude of Relative Error (RMMRE), the difference
between medium and low homogeneity was 19.4% (p= 0.032); and the difference between low and high was
17.1% (p = 0.051). This tendency was consistent with
Reliability of Median of Magnitude of Relative Error(RMdMRE), the difference between medium and low
homogeneity was 15.8% (p = 0.052); and thedifference between low and high was 22.9% (p =0.021).
The results also suggested that there would be an“optimized” threshold for CV to distinguish theunreliable estimates from reliable ones. Actually, if athreshold was put between the low and the medium
homogeneity, large differences can be observed in theestimation reliability. For instance, the difference of
RMMRE is 18.25% (nearly equal to the average of
19.4% and 17.1%); and the difference of RMdMRE is
19.35% (nearly equal to the average of 15.8% and22.9%).
A limitation of this study is that the hypothesis wastested based on just one dataset using one estimation
algorithm. To more fully verify the results, we need touse different datasets and algorithms.

8.

[2]

L. Angelis, and I. Stamelos, “A Simulation Tool for
Efficient Analogy Based Cost Estimation”, Empirical
Software Eng., vol.5, no.1, pp.35-68, 2000.

[3]

M. Auer, A. Trendowicz, B. Graser, E. Haunschmid,and S. Biffl, “Optimal Project Feature Weights in
Analogy-Based Cost Estimation: Improvement andLimitations”, IEEE Trans. on Software Eng., vol.32,
no.2, pp.83-92, 2006.

[4]

B.W. Boehm, “Software Engineering Economics”,
IEEE Trans. on Software Eng., vol.10, no.1, pp.4-21,
1984.

[5]

L.C. Briand, V.R. Basili, and W.M. Thomas, “A Pattern
Recognition Approach for Software Engineering Data
Analysis”, IEEE Trans. on Software Eng., vol.18, no.11,
pp.931-942, 1992.

[6]

S. Dutta, B. Wierenga, and A. Dalebout, “Case-Based
Reasoning Systems: From Automation to DecisionAiding and Stimulation”, IEEE Trans. on Knowledge
and Data Eng., vol.9, no.6, pp.911-922, 1997.

[7]

A. Gray, S. MacDonnell, and M. Shepperd, “Factors
Systematically Associated with Errors in Subjective
Estimates of Software Development Effort: The
Stability of Expert Judgment”, In Proc. of the 6th Int’l
Software Metrics Symposium, pp.216-227, 1999.

[8]

ISO/IEC 24570:2005: Software Engineering - NESMA
Functional Size Measurement Method v.2.2 Definitions and Counting Guidelines for the
Application of Function Point Analysis, 2005.

[9]

M. Jørgensen, and D.I.K. Sjøberg, “An Effort
Prediction Interval Approach Based on the Empirical
Distribution of Previous Estimation Accuracy”,
Information and Software Technology, vol.45, no.3,
pp.123-136, 2003.

[10] M. Jørgensen, and K. Moløkken-Østvold, “Reasons for
Software Effort Estimation Error: Impact ofRespondent Role, Information Collection Approach,and Data Analysis Method”, IEEE Trans. on SoftwareEng., vol.30, no.12, pp.993-1007, 2004.

Acknowledgements

This work has been done under the collaboration
between Nara Institute of Science and Technology andSoftware Engineering Center, Information-technology
Promotion Agency, Japan. The collaboration is
supported by the Comprehensive Development of eSociety Foundation Software program of the Ministryof Education, Culture, Sports, Science and Technology.

[11] A.L. Lederer, and J. Prasad, “Information Systems
Software Cost Estimating: A Current Assessment”,
Journal of Information Technology, vol.8, no.1, pp.2233, 1993.

9.

References[12] A.L. Lederer, and J. Prasad, “Causes of Inaccurate
Software Development Cost Estimates”, Journal of
Systems and Software, vol.31, no.2, pp.125-134, 1995.

[1]

D.M. Allen, “The Relationship between Variable
Selection and Data Augmentation and a Method for
Prediction”, Technometrics, vol.16, no.1, pp.125-127,[13] C. Lokan, T. Wright, P.R. Hill, and M. Stringer,“Organizational Benchmarking Using the ISBSG Data
Repository”, IEEE Software, vol. 18, no. 5, pp. 26-32,
2001.

391

[14] P.S. Myles, T. Gim, “Statistical Methods for
Anaesthesia and Intensive Care”, ButterworthHeinemann, 2000.
[15] T. Mukhopadhyay, S.S. Vicinanza, M.J. Prietula,“Examining the Feasibility of a Case-Based Reasoning
Model for Software Effort Estimation”, The
Management Information Systems Quarterly, vol.16,
no. 2, pp.155-171, 1992.
[16] K. Pearson, “On the Dissection of AsymmetricalFrequency Curves”, Philosophical Transaction ofRoyal Society, vol.A185, pp.71-110, 1894.
[17] D. Phan, D. Vogel, and J. Nunamaker, “The Search for
Perfect Project Management”, Computerworld, pp.95100, 1988.
[18] M. Shepperd, and C. Schofield, “Estimating Software
Project Effort Using Analogies”, IEEE Trans. on
Software Eng., vol.23, no.12, pp.76-743, 1997.
[19] Software Engineering Center, Information-Technology
Promotion Agency, Japan, “The 2005 White Paper on
Software Development Projects”, Nikkei Business
Publications, 2005 (in Japanese).
[20] G.H. Subramanian, and S. Breslawski, “An Empirical
Analysis of Software Effort Estimate Alterations”,
Journal of Systems and Software, vol.31, no.2, pp.135141, 1995.
[21] J.W. Tukey, “Exploratory Data Analysis”, AddisonWesley, 1977.
[22] N. Ohsugi, M. Tsunoda, A. Monden, T. Matsumura, K.
Matsumoto, and N. Kikuchi, “Using Cross-company
Data to Estimate the Effort of Software Development
Projects”, SEC journal, no.5, pp.16-25, 2006 (inJapanese).
[23] K. Toda, M. Tsunoda, A. Monden, and K. Matsumoto,“Characterizing Software Projects Unpredictable byEffort Estimation Models”, Technical Report of IEICE,
No.SS2005-74, pp.67-72, 2005 (in Japanese).
[24] M. Tsunoda, A. Monden, H. Yadohisa, N. Kikuchi, K.
Matsumoto, “Productivity Analysis of JapaneseEnterprise Software Development Projects”, In Proc. of2006 Int’l Workshop on Mining Software Repositories(MSR2006), pp.14-17, 2006.
[25] M. Van Genuchten, “Why is Software Late? An
Empirical Study of Reasons for Delay in Software
Development”, IEEE Trans. on Software Eng., vol.17,
no.6, pp.582-590, 1991.
[26] F. Walkerden and D.R. Jeffery, “An Empirical Study of
Analogy-Based Software Effort Estimation”, Empirical
Software Engineering, vol.4, no.2, pp.135-158, 1999.

392