Analyzing Risk Factors Affecting Project Cost Overrun

Masateru Tsunoda, Akito Monden,
Kenichi Matsumoto

Ryosuke Hatano, Toshihiko Nakano,
Yutaka Fukuchi

Graduate School of Information Science
Nara Institute of Science and Technology
Nara, Japan{masate-t, akito-m, matumoto}@is.naist.jp

Hitachi, Ltd., Tokyo, Japan{ryosuke.hatano.jn, toshihiko.nakano.yy,
yutaka.fukuchi.tt}@hitachi.com

Abstract—To prevent cost overrun of software projects, it iseffective to predict the project which has high risk of cost
overrun in the early phase of the project. In this paper, weclarify the risk factors which affect cost overrun. The risk
factors are denoted by the questions such as “Are thecustomer’s project goals clear?” The risk factors can be used
as independent variables of the cost overrun prediction model.
In the analysis, we used 290 projects’ data collected in asoftware development company. The dataset was stratified by
the project start time and the project size to eliminate theirinfluence, and relationships between risk factors and cost
overrun were analyzed with the correlation ratio. In addition,
we focused risk factors which have strong and stable
relationships to cost overrun, and analyzed them using the
Sharpe ratio based index. As a result, we identified some risk
factors which have relatively strong and stable relationships tocost overrun. After the analysis, we experimentally predicted
cost overrun projects by collaborative filtering, using the risk
factors as independent variables. The result suggested that cost
overrun projects can be predicted by the risk factors.

goals clear?” [6]) are used as independent variables for the
project result prediction model [4][10][13]. The model isbuilt from past projects’ data, and current project’s data is
input as independent variables to predict the project result.
The objective of the paper is to clarify the risk factors
which affect project cost overrun. They are used as
independent variables of the cost overrun prediction model.
We analyzed 290 projects’ data collected in a softwaredevelopment company. In the dataset, each risk factor was
evaluated by four-level Likert scale, and the degree of cost
overrun was determined based on difference between
estimated cost and actual cost. In the analysis, we examined
relationships between each risk factor and cost overrun.
When analyzing the dataset, we consider the change ofcharacteristic of the dataset over time, because the analysis
result shown in [2] suggests the characteristic varies overtime.
After the analysis, we predicted cost overrun projects,
using the factors which strongly relate cost overrun as
independent variables. We applied collaborative filtering topredict cost overrun projects. It is originally used for the item
(books or music) recommender system. Collaborative
filtering is based on k-nearest neighbor algorithm, as theanalogy based estimation method [9]. Roughly speaking,collaborative filtering finds projects similar to the target
project, and makes prediction based on values of dependent
variable of similar projects.
Our analysis results clarify which risk factors should becared especially. Controlling the factors will suppress theprobability of project cost overrun. Additionally, the results
enable project managers to predict cost overrun projects. In
what follows, Section II explains the dataset used in theanalysis. Section III describes the analysis of relationships
between risk factors and cost overrun. Section IV shows theresults of cost overrun project prediction. Section V
introduces related works. In the end, Section VI concludes
the paper with a summary.

Keywords-correlation ratio; Sharpe ratio; stratification; risk
management; collaborative filtering

I.

INTRODUCTION

Recently, software is widely used as a part ofinfrastructure of the our daily life such as banking system
and air traffic control system, while software size and cost(i.e. development effort) became extremely larger than ever.
As a result, one single overrun project can cause serious
damage to the profit of a software development company.
Therefore, prevention of cost overrun became extremely
important today.
One effective way to prevent cost overrun is to identify
the project which has high risk of cost overrun (project
failure) in the early phase of the project [4][10] so thatcountermeasures can be performed. To predict the project
result (project failure), discriminant methods such as linear
discriminant analysis or logistic regression has been used[4][10][13]. On a discriminant method, the project result is
set as the dependent variable, and its value (i.e. cost overrun
or not) is predicted from independent variables which are
known at prediction point of time. Usually, project
manager’s answers for questionnaires related to risk factors(for example, the question is “Are the customer’s projectII.

DATASETWe used risk evaluation data collected in a softwaredevelopment company in the 2000s. In the dataset,
evaluations of risk factors and the degree of cost overrun isrecorded for each software development project. The projects
mainly developed enterprise application software. The

TABLE I.
Identifier

Knowledge area

Upstream H10

Communication

Upstream O

Cost

Downstream H24

Cost

Upstream H46

Cost

Upstream H1

Customer

Upstream H72

Customer

RISK FACTORS USED IN THE ANALYSIS

Description [6][7][8]Are there minutes of reviews with the customer, and have they been agreed upon (approved) bymeans such as approval signatures?
Is effort estimated by the quantitative estimation tool?
How far can cost constraints be adjusted?
Has the size of systemization been estimated? In doing so, has the basis for the estimate beenrecorded?
Are requirements from customers of what they want to achieve clearly described in RFPs, etc?
Also, have the project members understood them?
If there is a need to assure current system functions, are current documents sufficiently maintained?

Upstream S3

Customer

Are the customer’s project goals clear?

Upstream H28

Human Resources

Have key personnel with required business knowledge been acquired?

Upstream S15

Integration

Upstream H14

Organization

Upstream S7Organization

Downstream H54

Risks

Are the deliverables and products for each task clear?
Does the project manager have experience appropriate for the scale and characteristics of the project?
If the project manager’s experience is insufficient, is there organizational support?
In the project organization, are the responsibility assignment of stakeholders including customers
clear, and are there any organizational deficiencies or concerns?
What is the project size?

Upstream S14

Scope

Upstream H71

Scope

Upstream H42

Scope

Midstream H12

Technology

Upstream S11, S12

Human
Time

Resources,

Has the feasibility of the requirements defined in the specifications been verified?
If the system connects to other systems developed by another company, is the responsibility
assignment clear?
Is there an agreement with the customer regarding the (contractual) handling of specification changes,
and are measures such as separate payment being practiced?
When using new programming language or technology, has the past record been confirmed on theapplication area? If there is no record, do you have the contingency plan for problems?
Are there sufficient human resources with required skills?
Has a plan for allocating personnel (in terms of quantity) been created?

dataset includes 290 projects and over 200 risk factors. The
risk factors were evaluated by a project manager at a certain
time such as the end of the design phase. We selected risk
factors which were evaluated until acceptance of order,
because we assume cost overrun prediction is performed at
the time. Additionally, we only chose risk factors which arealmost same as the factors which are already known to the
public (defined in [6][7][8]). This is because risk factors areindustrial secrets. As a result, we analyzed 17 risk factors
described in Table I. In the table, each identifier corresponds
to the identifier of the factors defined in [6][7][8]. The
knowledge area denotes the area of PMBOK (ProjectManagement Body of Knowledge) [5] to which each factor
is classified (The classification is written in [6][7][8]). Notethat only “Upstream O” is not defined in [6][7][8].
Each risk factor was evaluated by four-level Likert scale.
The levels are “high risk,” “middle risk,” “low risk,” and“unrelated”. If status of a project corresponded with adescription of a risk factor well, the risk factor was evaluatedas “high risk.” Similarly, if it did not correspond with thefactor at all, the factor was evaluated as “low risk.” Whencondition of a project was different from a risk factor, thefactor was evaluated as “unrelated.” For instance, when asystem did not connect to other systems developed byanother company, the risk factor “Upstream H71” wasevaluated as “Unrelated.” Some risk factors were evaluatedby three-level Likert scale (“high risk,” “low risk,” or“unrelated”).
Before analyzing, the evaluations of the risk factors(“high risk”, “middle risk”, “low risk”, and “unrelated”)
were converted to numerical values (4, 3, 2, and 1). Somerisk factors have missing values (i.e. a factor was notevaluated). The evaluations of the risk factors are originallyused for project management (They are not used for cost
overrun prediction).
Cost overrun is defined as the difference between
estimated cost and actual cost. It is signified by six ranks (1to 6), and small value means the difference was small (Wedo not know actual difference between estimated cost andactual cost, because they were not provided due toconfidentiality). Note that cost overrun does not relate toprofit well, because the profit is defined as the difference
between price and cost, and it is different from each project.
In the analysis, projects whose cost overrun was greaterthan four were treated as cost overrun projects, and otherprojects were treated as non cost overrun projects. We
assigned the value 1 to the cost overrun projects, and thevalue 0 to the non cost overrun projects. For discriminant
methods such as linear discriminant analysis are used to
predict cost overrun projects. Cost overrun projects are fairly
fewer than non cost overrun projects (Although cost overrun
projects defined in the paper are not failure project, we doCorrelation Ratio

0.35
0.30
0.25
0.20
0.15
0.10
0.05
0.00

All
Older
Newer

Figure 1. The correlation ratio of risk factors stratified by the project star time.

not show the actual rate of cost overrun projects because ofconfidential).
III.

RELATIONSHIPS BETWEEN RISK FACTORS AND COST
OVERRUNon the newer subset. On the contrary, the relationshipbetween Downstream H54 and cost overrun weakened on the
newer dataset. The result suggests the (learning) dataset
should be divided when cost overrun projects are predicted.

B. Analysis stratified by the project start time and the
project size
In Figure 1, Downstream H54 has the highest
relationship to cost overrun on whole dataset. This means
when the project size is large, the probability of cost overrun
becomes high. There is a probability that only the projectsize (Downstream H54) affects cost overrun, and other risk
factors have spurious relationships to cost overrun. So weanalyzed the relationships between risk factors and cost
overrun when the influence of the project size waseliminated. We stratified the dataset by the evaluation of
Downstream H54 (project size) and analyzed them.
Figure 2 shows the correlation ratio of each risk factor
SST(1) when projects are stratified by the project start time and theη2 =SSB
project size. In the figure, “Older-low” signifies thecorrelation ratio on the subset where projects were older andIn the equation, SSB is the sum of squares between groups
their project sizes were small (The evaluations of(signified by the nominal scale variable), and SST is the sum
Downstream H54 were “low risk”). Others such as “Newerof squares total.
High” are the same meaning. We omitted projects whose
We divided the dataset into two subsets in chronologicalevaluations of Downstream H54 were “unrelated,” because
order (based on the project start time), and calculated themost of their risk factors do not have relationships to costcorrelation ratio on whole dataset and the two subsets. The
overrun. To make the figure more visible, we cut the bar ofanalysis result shown in [2] suggests the characteristic varies
Upstream H72 on newer-high subset (The actual value isover time. Similarly, we assumed the characteristic of the0.71).
dataset was changed over time, because process
On some risk factors such as Upstream S3, theimprovement was performed during collecting the dataset,differences of the correlation ratio among subsets are
and it might affect the relationships between risk factors andrelatively small. That is, they have steady relationships tocost overrun.
cost overrun on any subsets. But other factors such as
Figure 1 shows the correlation ratio of each risk factor
Upstream H28 have unstable relationships. The former isstratified by the project start time. In the figure, “All,”
considered to be common relationships, but the letter is not.
“Older,” and “Newer” signify the correlation ratio on the
C. Analysis based on the Sharpe ratio based index
whole dataset, the older subset, and the newer subset. On
average, the strength of the relationships between the risk
Figure 2 signifies there are risk factors which have steadyfactors and cost overrun is not different between the older
relationships to cost overrun and factors which have unstable
subset and the newer subset (Both averages were 0.12).
relationships. We assumed that on the former factors, theHowever, some risk factors have large difference. So, theaverage of the correlation ratio is large and the variance isrelationships between the risk factors and cost overrun were
small among the subsets.
greatly changed over time. For example, the correlation ratioof Upstream S15 is small on the older subset, but it is large

A. Analysis stratified by the project start time
We analyzed relationships between risk factors and cost
overrun using the correlation ratio. As described in Section II,
risk factors were treated as ordinal scale variable and cost
overrun was treated as nominal scale. The correlation ratio isused to analyze the relationship between a nominal scalevariable and an ordinal scale. The value range of thecorrelation ratio is [0, 1], and the large value indicates there
is a strong relationship between the variables. The
correlation ratio η2 is calculated by:Correlation Ratio

0.50
0.45
0.40
0.35
0.30
0.25
0.20
0.15
0.10
0.05
0.00

Older-Low

Older-Mid

Older-High

Newer-Low

Newer-Mid

Newer-High

Figure 2. The correlation ratio of risk factors stratified by the project start time and the project size.

To identify the factors, we used the Sharpe ratio based
index (SRBI) [11]. The Sharpe ratio is originally used toevaluate performance of a portfolio (combined financial
products). It takes into account not only profit but also risk.
The value is low when profit is high but risk (standard
deviation) is also high. The Sharpe ratio s is calculated by:

Figure 3 illustrates the average, the standard deviation,and the SRBI of each risk factor. Top five risk factors whichhave large SRBI are Upstream H10, Upstream O, Upstream
H46, Upstream S3, and Upstream S15. From the result, weconcluded that they have relatively stable relationships tocost overrun. It is preferable to fulfill the following
conditions to avoid cost overrun.
• There are minutes of reviews with the customer, ands = (p - b) / r(2)
they have been agreed upon (approved) by means
such as approval signatures. (Upstream H10)
where p is profit rate of a portfolio, r is standard deviation of• Effort is estimated by the quantitative estimation tool.
the profit rate, and b is profit rate of risk-free asset. The(Upstream O)
SRBI c is calculated by:• The size of systemization has been estimated. In
doing so, the basis for the estimate has been recorded.
c = (a - m) / d(3)(Upstream H46)
• The customer’s project goals are clear. (Upstream
where a is the average of the index, d is the standard
S3)
deviation of it, and m is a baseline value whose function is•
The deliverables and products for each task are clear.
same as b in equation (2). Originally, in equation (2), when p(Upstream S15)
is smaller than b, the portfolio is regarded as useless.
Similarly, in equation (3), when the target index was
D. Analysis based on Precision
correlation coefficient and its value was 0.1, we considered it
We confirm the presence of the risk factor when its
as meaningless and set m to 0.1.
1.00
Average

SRBI

0.80
0.60
0.40
0.20
0.00
-0.20

Figure 3. The SRBI of risk factors.

S.D.

SRBI

Same as the precision, when calculating the recall of a risk
factor toward cost overrun, “cost overrun” was treated as
Actual value“The actual value is true,” and “high risk” was treated as
True
False
“The predicted value is true.” Also, the calculation of recall
True
TP
FP
Predicted
of a factor for non cost overrun is same as the precision.
value
False
FN
TN
Based on the analysis result shown in Figure 1, we
stratified the dataset by the project start time, and calculated
the precision and the recall (We did not stratified the dataset
evaluation is “high risk,” the project will be cost overrunby the project size because that makes the subset too small
certainly. (Note that this does not mean when its evaluation
for calculating the precision and the recall).
is not “high risk,” the project will be non cost overrun
Figure 4 shows the precision and the recall of each riskcertainly). Also, we confirm the presence of the risk factor
factor for cost overrun. As stated in Section II, cost overrun
when its evaluation is “unrelated” or “low risk,” the projectprojects were fairly fewer than non cost overrun projects, andwill not be cost overrun certainly. The risk factors are useful
therefore the precision and the recall are not high. Although
to predict project result (cost overrun or not) manually.
the precision of Upstream H46 is 100% on the older subset,
To identify the factors, we used the precision. Originally,
the number of the project is only one. So, the result is not
the precision is used to evaluate the accuracy of discriminantreliable. The precision of Upstream H1 and Upstream H71 is
methods. The precision is calculated by:relatively high on the older subset. Four projects out of fivewere cost overrun when the evaluation of Upstream H1 was
TP(4) “risk high,” and three projects out of four were cost overrun
Precision =TP + FP
when the evaluation of Upstream H71 was “risk high.”
However, the precision is not high on the newer subset. So,
Definitions of TP (true positive), FN (false negative), FP
we did not conclude they are notable risk factors.
(false positive), and TN (true negative) are denoted in Table
Figure 5 shows the precision and the recall of each risk
II. When calculating the precision of a risk factor for cost
factor for non cost overrun. The precision and the recall are
overrun, “cost overrun” was treated as “The actual value ishigh on average, because cost overrun projects were fairly
true,” and “high risk” was treated as “The predicted value isfewer than non cost overrun projects. There is no risk factor
true.” Similarly, when calculating the precision of a risk
whose precision is greater than 90%.
factor for non cost overrun, “non cost overrun” was treated
In the analysis, we did not find the risk factor when its
as “The actual value is true,” and “unrelated” and “low risk”
evaluation is “high risk,” the project will be cost overrun
were treated as “The predicted value is true.”certainly, and the factor when its evaluation is “unrelated” orAdditionally, for each risk factor, we confirm the rate
“low risk,” the project will not be cost overrun certainly.
that when a project was cost overrun, the factor of theevaluation was “high risk.” We used the recall to calculate
IV. COST OVERRUN PREDICTION
the rate. The recall is also used to evaluate the accuracy of
discriminant methods. The recall is calculated by:
A. Overview
We clarify prediction accuracy of cost overrun projects,
using the risk factors which have relatively strongTP(5) relationships to cost overrun. We used the risk factors whose
Recall =FP + FN
correlation ratio was equal to or greater than 0.1. We
predicted cost overrun projects using whole dataset and two

Precision / Recall

TABLE II.

100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%

DEFINITIONS OF TP, FN, FP, AND TN

Older-Precision

Newer-Precision

Older-Recall

Newer-Recall

Figure 4. The precision and the recall of each risk factor for cost overrunPrecision / Recall

100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%

Older-Precision

Newer-Precision

Older-Recall

Newer-Recall

Figure 5. The precision and the recall of each risk factor for non cost overrun.

subsets of the dataset divided by project start time, andcompared prediction accuracy of them. This is because theanalysis result in Figure 1 suggests the dataset should be
divided. We did not divide the dataset by the project size
because it has strong relationship to cost overrun as shown in
Figure 1. It means the risk factor of project size is effective
to predict cost overrun. The other reason is that small subset
depresses prediction accuracy.
We applied collaborative filtering for prediction, because
when the percentage of cost overrun and non cost overrun
projects are imbalanced, collaborative filtering is the mostappropriate prediction method [12].
B. Collaborative filtering
Originally, collaborative filtering is used for therecommender system which estimates users’ preferences torecommend items such as books or music. Collaborative
filtering presumes “Users who have similar preferences likesimilar items.”
Collaborative filtering uses m×n matrix shown in Table
III. In the matrix, Proji is i-th project, Qj is j-th independent
variable, vij is a value of Qj of Proji, and yi is the value of thedependent variable. We presume Proja is predicted project,and ŷ a is the predicted value of ya. Procedures ofcollaborative filtering consist of the three steps described
below.
Step 1 (normalization): Since a dependent variable andindependent variables have different ranges of value, thisstep makes the ranges [0, 1]. The value v´ij, normalized thevalue of vij is calculated by:vij′ =

Step 2 (similarity computation): This step computes
similarity Sim(Proja, Proji) between the predicted project pa
and other projects pi by:m

Sim(Proja , Proji ) =(6)

In the equation, max(Qj) and min(Qj) denote themaximum and minimum value of Qj respectively.

(7)

ah ih

h =1

m

∑ v′

m

2

∑ v′

ah

2

ih

h =1

h =1

The range of the value of Sim(Proja, Proji) is [0, 1].
Step 3 (computation of predicted value): The predictedvalue is computed by weighted average of the independent
variable of similar projects. Formally, the predicted value iscomputed by:∑Sim(Proj , Proj )( y′ − v′ )∑Sim(Proj , Proj )va′ +yˆ a′ =

a

h

h

h

(8)

h∈Sumprojectsahh∈Sumprojects

In the equation, Simprojects denotes the set of k projects(neighborhoods) which have top similarity with Proja. The
neighborhood size k affects prediction accuracy. The value
ˆ a is the normalized value of ŷ a . The value v h′ is they′average of v´ih included in Projh. On the recommendersystem, collaborative filtering uses users’ ratings for items.
However, some people tend to rate every item as high, andTABLE III.

vij − min (Q j )max (Q j ) − min (Q j )∑ v′ v′

Proj1
Proj2
…
Proji
…
Projm

Result
y1
y2
…
yi
…
ym

MATRIX USED BY COLLABORATIVE FILTERING
Q1
v11
v21
…
vi1
…
vm1

Q2
v12
v22
…
vi2
…
vm2

…
…
…
…
…

Qj
v1j
v2j
…
vij
…
vmj

…
…
…
…
…

Qn
v1n
v2n
…
vin
…
vmn

on the other hand, some do as low. Hence, this equation uses
the difference from average of each people’s rating. We
applied this algorithm to predict the project result, because
our dataset seems to have similar characteristic.
In the experiment, we skipped Step 1 because the rangeof the value of each risk factor is the same as each other. The
number of neighborhoods k was decided based on apreliminary analysis.
C. Evaluation criterion
We used area under the curve (AUC) [1] as theevaluation criterion of cost overrun prediction. AUC isrecently used to evaluate discriminant methods in software
engineering researches, for it is more appropriate criterion
for discriminant methods than other criteria like F1 score [3].
The value range of AUC is [0, 1], and higher AUC meansthat prediction accuracy of the method is high. When AUC issmaller than 0.5, the prediction result is same as random
prediction. AUC is defined as the area under the receiver
operating characteristic (ROC) curve. ROC curve is drawn
by changing threshold and calculating true positive rate andfalse positive rate. These rates are calculated by:

TABLE IV.

AUC ON EACH DATASET

Learning data

AUC

Older subset

0.68

Newer subset

0.56

Whole dataset (Older; divided after prediction)

0.65

Whole dataset (Newer; divided after prediction)

0.54

were divided by the project start time after the prediction.
Prediction results based on the older subset and the newer
subset show higher accuracy than whole dataset. Although
the difference is not large, the result suggests that data
stratification by the project start time is effective to enhancecost overrun prediction accuracy.
AUC on each dataset is greater than 0.5, and therefore
cost overrun projects can be predicted by the risk factors.
However, AUC is not high, and other risk factors are needed
to improve the prediction accuracy.

V. RELATED WORK
There are some researches which analyzed relationships
TP
betweenrisk factors and project results, and predicted project(9)
True Positeve Rate =results. Takagi et al. [10] defined confused projects based on
TP + FN
the ratio of the actual resultant cost and the planned cost, andanalyzed 32 projects collected in a software development
company in 1990s. They pointed out risk factors aboutFP(10) estimations and planning are important. Our analysis result
False Positive Rate =FP + TNand their result are similar in regard to the influence ofUpstream S15. However, they did not analyze other risk
High true positive rate and false positive rate means highfactors which showed stable relationships in Figure 3
accuracy. But there is tradeoff between them, and they
(Upstream H10, Upstream O, Upstream H46, and Upstream
depend on a threshold. For example, if prediction is done byS3).
logistic regression and the threshold is set as 0, true positive
Procaccino et al. [4] defined success project based onrate is very high but false positive rate is very low. AUC canasking of developers, and analyzed 42 projects collectedevaluate performance of discriminant methods independently
from 21 developers in 1999. They identified some risk
from the threshold.
factors related to success of the project. The major difference
between our research and their research is they did not
D. Experimental Procedure
analyze risk factors which showed stable relationships in
We predicted the project result (Cost will be overrun or
Figure 3.
not) according to the following procedure. The procedure
Also, Verner et al. [13] (co-author of [4]) analyzed
was applied to whole dataset, older subset, and newer subset.
relationships between success of the project (The definition
1. The risk factors whose correlation ratio is smaller
is same as [4]) and risk factor. They collected datasets fromthan 0.1 are removed from the dataset.
the United States and Australia, and analyzed them. They
2. In the dataset, a project is regarded as a test data
pointed out the initial effort estimation is important for(ongoing project and the project result is unknown),
success of the project. However, they did not analyze risk
and other projects are regarded as the learning
factors which affect estimation accuracy. We analyzed thedataset (finished project and the project result isrisk factors (Upstream O and Upstream H46), and that is the
known).
major difference between our research and their research.
3. The project result of the test data is predicted bycollaborative filtering based on the learning dataset.
VI. CONCLUSIONS
4. For each project in the dataset, Step 2 to 3 are
In this paper, we clarified risk factors which haverepeated (leave-one-out cross-validation).
relatively strong relationships to cost overrun of the software
5. The evaluation criterion (AUC) is computed.
development project. We analyzed software development
project dataset, removing effects of the project start time andE. Prediction results
the project size. The analysis results suggested there are
Table IV shows AUC on each dataset. To makesome risk factors which have relatively strong and stablecomparison easy, predicting results based on whole dataset

relationships to cost overrun. Project managers should carethe followings especially, to avoid cost overrun.
• Reviews with the customer and approval.
• The quantitative estimation by the tool.
• The estimation of the systemization size and thebasis for it.
• The clarity of customer’s project goals.
• The clarity of deliverables and products for each task.
Based on the above analysis, we selected some risk
factors, and used them as independent variables of cost
overrun prediction. We applied collaborative filtering topredict cost overrun, and experimental result showed thatcost overrun projects can be predicted using the risk factors.
Our future work is identifying other risk factors which are
more effective to predict cost overrun projects.
ACKNOWLEDGMENT
This work is being conducted as a part of the StagE
project, The Development of Next-Generation IT
Infrastructure, and Grant-in-aid for Young Scientists (B),
22700034, 2010, supported by the Ministry of Education,
Culture, Sports, Science and Technology. We are deeply
grateful to people who cooperated for data collection. Also,
we would like to thank Dr. Naoki Ohsugi for offering thecollaborative filtering tool.
REFERENCES
[1]

[2]

[3]

J. Hanley, and B. McNeil, “The meaning and use of the area under areceiver operating characteristic (ROC) curve,” Radiology, no.143,
pp.29-36, 1982.
B. Kitchenham, S. Pfleeger, B. McColl, and S. Eagan, “An empirical
study of maintenance and development estimation accuracy,” Journalof Systems and Software, vol.64, no.1, pp. 57-77, 2002.
S. Lessmann, B. Baesens, C. Mues, and S. Pietsch, “Benchmarking
Classification Models for Software Defect Prediction: A Proposed

Framework and Novel Findings,” IEEE Transactions on Software
Engineering, vol.34, no.4, pp.485-496, 2008.
[4] J. Procaccino, J. Verner, S. Overmyer, and M. Darter, “Case study:
factors for early prediction of software development success,”
Information and Software Technology, vol.44, no.1, pp.53-62, 2002.
[5] Project Management Institute, A Guide to the Project Management
Body of Knowledge: (PMBOK Guide). p.459, Project Management
Institute, 2008.
[6] Software Engineering Center, Information-Technology Promotion
Agency, Japan, “MIERUKA (Visualization)” of IT Projects:
Upstream Process, Software Engineering Center, InformationTechnology Promotion Agency, Japan, 2010. http://www.ipa.go.jp/
english/sec/reports/20100507b/20100507b_Upsteam.pdf[7] Software Engineering Center, Information-Technology Promotion
Agency, Japan, “MIERUKA (Visualization)” of IT Projects:
Midstream Process, Nikkei Business Publications, 2008 (in Japanese).
[8] Software Engineering Center, Information-Technology Promotion
Agency, Japan, “MIERUKA (Visualization)” of IT Projects:
Downstream Process, Software Engineering Center, InformationTechnology Promotion Agency, Japan, 2010. http://www.ipa.go.jp/
english/sec/reports/20100507/20100507b_Downsteam.pdf[9] M. Shepperd, and C. Schofield, “Estimating software project effort
using analogies,” IEEE Transactions on Software Engineering, vol.23,
no.12, pp.736-743, 1997.
[10] Y. Takagi, O. Mizuno, and T. Kikuno, “An Empirical Approach toCharacterizing Risky Software Projects Based on Logistic Regression
Analysis,” Empirical Software Engineering, vol.10, no.4, pp.495-515,
2005.
[11] M. Tsunoda, A. Monden, and K. Matsumoto, “Sharpe Ratio Based
Index for Building Fault Prediction Model,” Supplemental Proc.
International Symposium on Software Reliability Engineering(ISSRE 2011), Vol.1, No.6, pp.1-2, 2011.
[12] M. Tsunoda, A. Monden, J. Shibata, and K. Matsumoto, “Empirical
Evaluation of Cost Overrun Prediction with Imbalance Data,” Proc.
International Conference on Computer and Information Science (ICIS
2011), pp.415-420, 2011.
[13] J. Verner, W. Evanco, and N. Cerpa, “State of the practice: An
exploratory analysis of schedule estimation and software projectsuccess prediction,” Information and Software Technology, vol.49,
no.2, pp.181-193, 2007.