Empirical Evaluation of Cost Overrun Prediction with Imbalanced Dataset

Masateru Tsunoda, Akito Monden, Jun-ichiro Shibata and Ken-ichi Matsumoto
Graduate School of Information Science
Nara Institute of Science and Technology
Nara, Japan{masate-t, akito-m, junichiro-s, matumoto}@is.naist.jp

Abstract—To prevent cost overrun of software projects, it isnecessary for project managers to identify projects which havehigh risk of cost overrun in the early phase. So far,discriminant methods such as linear discriminant analysis andlogistic regression have been used to predict cost overrun
projects. However, accuracy of discriminant methods often
becomes low when a dataset used for predict is imbalanced, i.e.
there exists a large difference between the number of cost
overrun projects and non cost overrun projects. In this paper,
we compared accuracy of linear discriminant analysis, logistic
regression, classification tree, Mahalanobis-Taguchi method,and collaborative filtering, by changing the percentage of cost
overrun projects in the dataset. The result showed thatcollaborative filtering was highest accuracy among fivemethods. When the number of cost overrun projects and noncost overrun was balanced in the dataset, linear discriminant
analysis was second highest accuracy, and when it was notbalanced, Mahalanobis-Taguchi method was second highest
among five methods.
Keywords-biased data; failure prone project; Collaborative
Filtering; Mahalanobis-Taguchi method; risk managementI.

INTRODUCTION

Recently, software is widely used as a part ofinfrastructure of the our daily life such as banking system
and air traffic control system, while software size and cost(i.e. development effort) became extremely larger than ever.
As a result, one single overrun project can cause serious
damage to the profit of a software development company.
Therefore, prevention of cost overrun became extremely
important today.
One effective way to prevent cost overrun is to identify
the project which has high risk of cost overrun (project
failure) in the early phase of the project [10][15] so thatcountermeasures can be performed. To predict the project
result (project failure), discriminant methods such as linear
discriminant analysis or logistic regression has been used[10][15][16]. On a discriminant method, the project result is
set as dependent variable, and its value (i.e. cost overrun ornot) is predicted from independent variables which are
known at prediction point of time. Usually, project
manager’s answers for questionnaires related to risk factors(for example, the question is “Insufficient explanation of therequirements” [15]) are used as independent variables for
project result prediction model [10][15][16]. The model isbuilt from past projects’ data, and current project’s data is
input as independent variables to predict the project result.
However, accuracy of discriminant methods often
becomes low when imbalanced dataset is used for prediction[5]. The imbalanced dataset means that there exists a large
difference between the number of cost overrun projects andnon cost overrun projects. For example, in the company
whose organizational maturity level is high (e.g. CMMI
(Capability Maturity Model Integration) level is over 2),
there would be less cost overrun projects, and that makes thepercentage of cost overrun projects low.
In this paper, we focus on Mahalanobis-Taguchi method
and collaborative filtering to apply cost overrun prediction.
Mahalanobis-Taguchi method is used as one of thetechniques for quality control of the manufacturing industry.
It builds a model using only normal cases (i.e. non cost
overrun projects), and predicts the project result based on the
distance from the normal case group. Therefore, it isexpected that the model is not affected by imbalance of thedataset. On the other hand, collaborative filtering is
originally used for the item (books or music) recommendersystem. Collaborative filtering is based on k-nearest neighboralgorithm, as the analogy based estimation method [13].
Roughly speaking, collaborative filtering finds projects
similar to the target project, and makes prediction based onvalues of dependent variable of similar projects. We applied
it to cost overrun prediction since the dataset used forprediction is similar to the dataset treated by collaborative
filtering.
We analyzed accuracy of discriminant methods when thepercentage of cost overrun projects and that of non cost
overrun are imbalanced. In the experiment, we changed thepercentage of cost overrun projects by deleting cost overrun
projects in the dataset whose data was collected in a softwaredevelopment company, and predicted the project result withlinear discriminant analysis, logistic regression, classification
tree, Mahalanobis-Taguchi method, and collaborative
filtering. The result of the experiment makes practitioners
choose discriminant methods more appropriately.
In what follows, Section II explains discriminant
methods used in the experiment. Section III describes
procedure of the experiment, the evaluation criterion of themethod, and the dataset used in the experiment. Section IVshows results of the experiment and discusses it. Section V
introduces related works. In the end, Section VI concludes
the paper with a summary.

DISCRIMINANT METHODS

The discriminant method builds a prediction model using
a dataset which includes finished projects whose dependent
variable is already known (i.e. cost overrun or not). The
result of an unfinished project is predicted by the model.
We evaluated accuracy of five types of discriminant
methods for predicting the project result. Linear discriminant
analysis, logistic regression and classification tree are widelyused as discriminant methods in the software engineering
field [6][9][15]. In addition, we applied collaborative
filtering and Mahalanobis-Taguchi method to predicting the
project result, for they are expected to fit to the dataset.
A. Linear discriminant analysis
Linear discriminant analysis makes a line which divides adataset into two groups based on a dependent variable. The
model of the line is:

Independent variable 2

II.

4
3
Non cost overrun project2

Cost overrun project

1

0
0

1
2
3
4
Independent variable 1

Figure 1. An example of the model of Mahalanobis-Taguchi method.

method assumes that although normal cases (projects) aresimilar to other normal cases, abnormal cases are not similar
to each other because they became abnormal cases fory  a1 x1  an xn  b different reasons. That is, Mahalanobis-Taguchi method
assumes that when cost overrun is not occurred, the reason is
In the model, y is a discriminant score, xn are independent
similar to other non cost overrun projects, but when costvariables, an are regression coefficients, and b is an intercept.
overrun is occurred, the reason of cost overrun is different
The dependent variable is predicted by whether thefrom other cost overrun projects. This assumption was
discriminant score is plus or minus.
inspired by the sentence in Anna Karenina, a novel written
by Tolstoy. It said “Happy families are all alike; every
B. Logistic regression
unhappy family is unhappy in its own way.” Based on the
Logistic regression predicts a dependent variable based
assumption, Mahalanobis-Taguchi method builds a model
on a logistic function. The model of logistic regression is:using only normal cases (i.e. non cost overrun projects).
Mahalanobis-Taguchi method is used in few software
1
engineering researches [1].
y( a1x1 an xx b )Mahalanobis-Taguchi method predicts a case (project) as
1

e



abnormal one (cost overrun project) when the case is distantfrom a normal case group (non cost overrun project group).
In the model, y indicates probability of the dependent
The distance is calculated by:variable, xn are independent variables, an are regression
coefficients, and b is an intercept. The dependent value is m  mi  maj  m j 
1predicted as probability. For example, when the value y is
Da2   rij  ai
0.7, probability of that predicted project will belong to one
k ij 
si
s
j




group is 70%.
C. Classification tree
Classification tree predicts a dependent variable by a tree
structure model which has leafs and nodes. Each leaf
indicates the predicted value of the dependent variable, andeach node has a condition related to one of independent
variables. Based on independent variables, a path on themodel is chosen, and it predicts the dependent variable.
There are some algorithms to build classification tree.
For example, CART (classification and regression trees)algorithm uses Gini index as a model building criterion, andID3 (Iterative Dichotomiser 3) and C4.5 algorithms use
information gain. In this paper, we used CART implemented
on R [12].
D. Mahalanobis-Taguchi method
Mahalanobis-Taguchi method [14] was proposed by
Taguchi, and it is used as one of the techniques for quality
control of the manufacturing industry. Mahalanobis-Taguchi

In the equation, Da indicates Mahalanobis distance ofpredicted project, k is the number of independent variables inthe dataset, rij is the inverse correlation matrix, mai is thevalue of i-th independent variable of the predicted project, si
is the standard deviation of i-th independent variable, and miis the average of i-th independent variable. Values of rij,
si ,and mi are calculated based on normal cases in the dataset.
If Da is greater than a certain threshold, the project ispredicted as a cost overrun project. Fig. 1 illustrates anexample of the model of Mahalanobis-Taguchi method. In
the figure, the border line indicates the threshold ofMahalanobis distance. If predicted project is out of theborder line area, the project is predicted as cost overrun
project.
E. Colaborative filtering
Originally, collaborative filtering is used for therecommender system which estimates users’ preferences torecommend items such as books or music. Collaborative
filtering presumes “Users who have similar preferences likesimilar items.” Few software engineering researches usedcollaborative filtering for prediction [8].
Collaborative filtering uses m×n matrix shown in Table 1.
In the matrix, Proji is i-th project, Qj is j-th independent
variable, vij is a value of Qj of Proji, and yi is the value of thedependent variable. We presume Proja is predicted project,and ŷa is the predicted value of ya. Procedures ofcollaborative filtering consist of the three steps described
below.
Step 1 (normalization): Since a dependent variable andindependent variables have different ranges of value, thisstep makes the ranges [0, 1]. The value v´ij, normalized thevalue of vij is calculated by:

TABLE I.

Proj1
Proj2
…
Proji
…
Projm

Result
y1
y2
…
yi
…
ym

MATRIX USED BY COLLABORATIVE FILTERING

…
…
…

Q1
v11
v21
…
vi1
…
vm1

Q2
v12
v22
…
vi2
…
vm2

III.

EXPERIMENT

…
…

Qj
v1j
v2j
…
vij
…
vmj

…
…
…
…
…

Qn
v1n
v2n
…
vin
…
vmn

A. Overview
In the experiment, to clarify proper discriminant methods
for predicting the project result, we evaluated accuracy ofvij  min Q j vij 
discriminant methods when the percentage of cost overrun
max Q j   min Q j  projects and that of non cost overrun were imbalanced.
Using 28 projects data collected in a software development
company, we changed the percentage of cost overrun
In the equation, max(Qj) and min(Qj) denote theprojects by deleting projects in the dataset from 50.0% (14maximum and minimum value of Qj respectively.
cost overrun projects) to 6.7% (1 cost overrun project), andStep 2 (similarity computation): This step computes
applied discriminant methods. The Methods used in thesimilarity Sim(Proja, Proji) between the predicted project pa
experiment were linear discriminant analysis, logistic
and other projects pi by:regression, classification tree, Mahalanobis-Taguchi method,and collaborative filtering.
m v  Q v  Q  B. Dataset
We used the questionnaire about the software project asvah  Qh2  vih  Qh2
thedataset for prediction. Project data in the dataset were
h 1
h 1
collected in the 2000s. The questionnaire is originally used
for project management. Although details of the
In the equation, Qh is average of Qh based on v´ij. With
questionnaire does not described due to confidential, it isthe equation, the value v´ij which is higher than averagesimilar to the questionnaire shown in a software projectshows positive values, and lower than average showsmanagement guidebook [4], which consists of questions
negative values to sharpen differences between projects. The
related to 9 knowledge areas of PMBOK (Project
range of the value of Sim(Proja, Proji) is [-1, 1].
Management Body of Knowledge) [11]. Similar
Step 3 (computation of predicted value): The predictedquestionnaires are also used in other project result prediction
value is computed by weighted average of the independent
researches [10][15][16]. For example, one of the questions isvariable of similar projects. Formally, the predicted value is“If new or unexperienced technologies are used in the project,computed by:is the project plan sufficient to cope with them? [4]” (NoteSim Proja , Proji  

h 1

ah

h

m

ih

h



m

that it is not entirely equal to the question in our dataset).
Questions are rated as “high risk”, “middle risk”, “low risk”, or “unrelated” based on probability of project failure. If newhSumprojectsyˆ a technology is used but the project plan is insufficient, theahhSumprojects
question is rated as “high risk”. If new technology is not used,
it is rated as “unrelated”.
In the equation, Simprojects denotes the set of k projects
We used these questions as independent variables of(neighborhoods) which have top similarity with Proja. The
prediction models. Only questions whose answers were clear
neighborhood size k affects prediction accuracy. The value
at early phase of the project were used as independentyˆ a is the normalized value of ŷa . The value vh is thevariables. Ratings “high risk”, “middle risk”, “low risk”, oraverage of v´ih included in Projh. On the recommender“unrelated” were converted to numerical values (4, 3, 2, andsystem, collaborative filtering uses users’ ratings for items.
1) before applying discriminant methods.
However, some people tend to rate every item as high, andCost overrun was set as the dependent variable. Weon the other hand, some do as low. Hence, this equation uses
defined cost overrun as overrun of actual cost from estimateddifference from average of each people’s rating. We applied
cost. Projects whose cost overrun were greater than certain
this algorithm to predict the project result, because ourthreshold were defined as cost overrun projects, and values
dataset seems to have similar characteristic.
of their dependent variable were set as 1. Other projects wereva 

 Sim( Proj , Proj )( y  v )  Sim( Proj , Proj )
a

h

h

h

defined as non cost overrun projects, and values of theirdependent variable were set as 0. Note that the threshold is
not disclosed in this paper because of confidential. Although
there are more than 100 projects in the dataset, cost overrun
projects are fairly fewer than non cost overrun projects.
Therefore, we randomly extracted projects from the dataset
to adjust the balance of cost overrun and non cost overrun
projects.
The dataset contains 120 questions. When the number of
independent variables is much greater than the number ofcases (projects), it is difficult to build a prediction model
appropriately (curse of dimensionality). So when correlation
coefficient between the project result and the question was
greater than or equal to 0.2, we used the question as theindependent variable. Moreover, we eliminated questions
which have missing values, to avoid influence of them. As aresult, 7 questions were selected for independent variables.
C. Evaluation criterion
We used area under the curve (AUC) [2] as theevaluation criterion of discriminant methods. AUC isrecently used to evaluate discriminant methods in software
engineering researches, for it is more appropriate criterion
for discriminant methods than other criteria like F1 score [7].
The value range of AUC is [0, 1], and higher AUC meansthat prediction accuracy of the method is high. AUC isdefined as the area under the receiver operating characteristic(ROC) curve. ROC curve is drawn by changing thresholdand calculating true positive rate and false positive rate.
These rates are calculated by:True Positeve Rate TP TP  FNTABLE II.

DEFINITIONS OF TP, FN, FP, AND TN

Predicted
value

True
False

Actual value
True False
TP
FP
FN
TN

2.

A test dataset was made in the same way (the test
dataset did not include as same projects as thelearning dataset).
3. Prediction models of five discriminant methods werebuilt using the learning dataset.
4. Each model was applied to the test dataset to predictthe project result, and the evaluation criterion (AUC)
was computed.
5. One cost overrun project was randomly deleted fromthe learning dataset.
6. Step 3 to 5 were repeated until the number of cost
overrun projects in the learning dataset was equal to1.
The percentage of cost overrun projects was fixed as
50.0% in the test dataset. When a model of MahalanobisTaguchi method was built, step 5 and 6 were not performed
because Mahalanobis-Taguchi method does not use cost
overrun project to build the model, and hence accuracy ofthe model is not affected by the percentage. The
neighborhood size of collaborative filtering was set as 5.
When using linear discriminant analysis and logistic
regression, both variable selection model and non variableselection model were built, because these discriminant
methods have commonly-used variable section method.
IV.

RESULTS AND DISCUSSION

Results of the experiment are shown in Fig. 2. In the
figure, the virtual axis indicates AUC and the horizontal axis
FP  indicates the number of cost overrun projects. AUC isFalse Positive Rate 
average of 10 results of the experiment. Regardless of theFP  TN
number of cost overrun projects, collaborative filtering
showed highest accuracy among five discriminant methods.
Definitions of TP (true positive), FN (false negative), FP
Table 3 shows one of 10 prediction results by(false positive), and TN (true negative) are shown in Table 2.
collaborative filtering when number of cost overrun projects
Although high true positive rate and false positive rate meansin the learning dataset was 1 (AUC was 0.77). As shown inhigh accuracy, there is tradeoff between them, and they
the table, collaborative filtering chose similar projects
depend on a threshold. For example, if prediction is done byappropriately. Although projects P27 and P28 did not havelogistic regression and the threshold is set as 0, true positive
similar projects whose actual values were 1, predicted values
rate is very high but false positive rate is very low. AUC canwere 1. This is because when average of independent
evaluate performance of discriminant methods independently
variables is high, predicted value of the dependent variable isfrom the threshold.
also high by (6). Note that even if average of independent
D. Exprrimantal Procedurevariables is high, predicted value of the dependent variable
can be low as shown in Table 4. Equation (6) reflects the
We changed the percentage of cost overrun and non cost
assumption that in particular projects, dependent variable canoverrun projects, and predicted the project result according
be low in spite of high average of independent variables.
to the following procedure. To avoid biased results, theActually, average values of independent variables of P08,
procedure was repeated 10 times.
P10, and P11 were same as P28, but predicted values were
1. 14 cost overrun projects and 14 non cost overrun
different from P28.
projects were randomly selected from the dataset to
The reason of high accuracy of collaborative filtering
make a learning dataset (i.e. the learning dataset
would be that both our dataset and a dataset used by acontains 28 projects).
recommender system based on collaborative filtering have

0.85

TABLE III.

0.80
0.70
0.65

AUC

0.75

0.60

0.55
0.50

0.45
14 13 12 11 10 9 8 7 6 5 4 3 2
The number of cost overrun projects

1

Linear discriminant analysis
Linear discriminant analysis (variable selection)
Logistic regression
Logistic regression (variable selection)
Classification tree
Mahalanobis-Taguchi method
Collaborative filtering
Figure 2. Relationship between accuracy and the number of cost
overrun projects.

similar characteristics (they are rating data). Moreover,accuracy of the method kept high when the percentage of
cost overrun projects got low. Collaborative filtering does
not build a model before prediction but uses preparedequation (lazy learning). As a result, when collaborative
filtering fits well to a dataset, accuracy is not considered tobe affected very much by imbalance of the dataset. Therefore,
collaborative filtering is most suitable for predicting the
project result, no matter what the percentage of cost overrun
projects is.
When the percentage of cost overrun projects and noncost overrun projects was balanced, accuracy ofMahalanobis-Taguchi method was lower. One of the reasons
may be that Mahalanobis-Taguchi method does not use cost
overrun projects to build a prediction model, andconsequently, available information amount is less than otherdiscriminant methods. However, when the percentage of cost
overrun projects was lower than 30.0% (6 cost overrun
projects), accuracy of Mahalanobis-Taguchi method wassecond among five discriminant methods. This would be
because other discriminant methods were overly affected bynon cost overrun projects, and as a result, accuracy ofMahalanobis-Taguchi method was comparatively higher.
Although Mahalanobis-Taguchi method is not fit well for
predicting the project result, it can be a candidate ofprediction methods when the dataset is imbalanced.
Accuracy of classification tree was the lowest in thediscriminant methods except the percentage of cost overrun
projects was 30.0% (6 cost overrun projects). Thus,
classification tree is not suitable for project result prediction.
Also, classification tree failed to build a model when thepercentage of cost overrun projects was low. When the

P01
P02
P03
P04
P05
P06
P07
P08
P09
P10
P11
P12
P13
P14
P15
P16
P17
P18
P19
P20
P21
P22
P23
P24
P25
P26
P27
P28

PREDICTION RESULTS BY COLLABORATIVE FILTERING
Predicted
value
0
0
1
0
1
0
0
0
0
0
0
1
1
0
1
1
1
1
1
0
0
1
0
1
1
0
1
1

Actual
value
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
1
1
1
1
1
1

0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
0
0
0
0
0
0
0
0
0
0
0

Actual value ofsimilar projects
0
0
0
0
0
0
1
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
1
0
0
0
0
0
0
0
1
0
0
0
0
0
0
1
0
0
1
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
1
0
0
0
1
0
0
0
0
0
0
0
0
0

0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

percentage of cost overrun projects was lower than 39.1% (9cost overrun projects), classification tree failed to build amodel in some cases of 10 repeated experiments, and when
the percentage was lower than 30.0%, it failed to build
models in all cases. The result means that classification tree
is greatly affected by imbalance of a dataset.
Accuracy of linear discriminant analysis was higher thanlogistic regression at any percentage of cost overrun projects.
In linear discriminant analysis models, accuracy of models
applied variable selection was almost higher than notapplied. In logistic regression models, when the percentage
of cost overrun projects was high, accuracy of models
applied variable selection was lower than not applied, butwhen the percentage was low, accuracy of models applied
variable selection was higher. So affects of the percentage of
cost overrun projects for variable selection is considered tobe different for discriminant method types.
V.

RELATED WORK

There are some researches about project result prediction
such as cost overrun [10][15][16]. For instance, Takagi et al.
[15] proposed delivery delay project prediction method using
logistic regression and a questionnaire about the project.
However, these researches did not compare accuracy of
discriminant methods, changing the percentage of failure
projects in a dataset, and therefore they did not clarify which
method is better when a dataset is imbalanced.
In the software engineering field, there are few
researches using Mahalanobis-Taguchi method orcollaborative filtering. Aman et al. [1] proposed theprediction method which identifies program modules whoseTABLE IV.

RELATIONSHIP BETWEEN PREDICTED VALUEAND SIMILAR PROJECTS

Similar projects(Learning dataset)
Average ofDependent
independent
variablevariable
Low
High
Low
Low

Predicted project(Test dataset)
Average of
Predicted
independent dependent
variablevariable
High
Low
High
High

modification effort is high by Mahalanobis-Taguchi method.
Motomura et al. [8] proposed cost overrun project prediction
using collaborative filtering. Again, these researches did notclarify accuracy of discriminant methods when a dataset isimbalanced.
As contrasted to Mahalanobis-Taguchi method, positive
unlabeled learning builds a prediction model without
negative cases (i.e. use only cost overrun projects). Hata et al.
[3] proposed applying positive naive Bayes to predict fault
prone modules. One of our future research issues is to apply
positive naive Bayes to project result prediction.
Kamei et al. [5] proposed applying oversampling to an
imbalanced dataset before predicting fault prone modules.
Oversampling duplicates one group whose number of cases
is smaller than the other group, to align imbalance of thedataset. The other future issue of our research is applying
oversampling and comparing accuracy of discriminant
methods, changing the percentage of cost overrun projects.
VI.

CONCLUSIONSIn this paper, we compared accuracy of discriminant
methods, changing the percentage of cost overrun projects.
We predicted cost overrun projects by linear discriminant
analysis, logistic regression, classification tree, MahalanobisTaguchi method, and collaborative filtering, using thequestionnaire about the software project. The result showedthat collaborative filtering was highest accuracy among five
discriminant methods. When the dataset was not very
imbalanced, linear discriminant analysis was second highest
in the methods, and when it is imbalanced, MahalanobisTaguchi method was second highest. Classification tree was
not fitted to project result prediction. Our future works are to
apply positive naive Bayes and oversampling to predict the
project result, changing the percentage of cost overrun
projects.
ACKNOWLEDGMENT
This work is being conducted as a part of the StagE
project, The Development of Next-Generation IT
Infrastructure, and Grant-in-aid for Young Scientists (B),
22700034, 2010, supported by the Ministry of Education,
Culture, Sports, Science and Technology. We are deeply
grateful to Mr. Yutaka Fukuchi (Hitachi Ltd.), Mr. Tetsuya

Yonemitsu (Hitachi Ltd.), and people who cooperated fordata collection. Also, we would like to thank Dr. Naoki
Ohsugi for offering the collaborative filtering tool.
REFERENCES
H. Aman, N. Mochiduki, and H. Yamada, “A Model for Detecting
Cost-Prone Classes Based on Mahalanobis-Taguchi Method,” IEICE
transactions on information and systems, vol.E89-D, no.4, pp.13471358, 2006.
[2] J. Hanley, and B. McNeil, “The meaning and use of the area under areceiver operating characteristic (ROC) curve,” Radiology, no.143,
pp.29-36, 1982.
[3] H. Hata, O. Mizuno, and T. Kikuno, “Application of Machine
Learning Without Negative Examples to Fault-Prone Module
Detection,” Proc. Software Engineering Symposium 2009 (SES 2009),
pp.133-138, 2009 (in Japanese).
[4] IPA/SEC Japan, Visualization of IT Project (lower process edition).
p.211, Nikkei Business Publications, 2006.
[5] Y. Kamei, A. Monden, S. Matsumoto, T. Kakimoto, and K.
Matsumoto, “The Effects of Over and Under Sampling on FaultProne Module Detection,” Proc. International Symposium on
Empirical Software Engineering and Measurement (ESEM 2007),
pp.196-204, 2007.
[6] T. Khoshgoftaar, and N. Seliya, “Comparative Assessment ofSoftware Quality Classification Techniques: An Empirical Case
Study,” Empirical Software Engineering, vol.9, no.3, pp.229-257,
2004.
[7] S. Lessmann, B. Baesens, C. Mues, and S. Pietsch, “Benchmarking
Classification Models for Software Defect Prediction: A Proposed
Framework and Novel Findings,” IEEE Transactions on Software
Engineering, vol.34, no.4, pp.485-496, 2008.
[8] T. Motomura, T. Kakimoto, M. Tsunoda, N. Ohsugi, A. Monden, andK. Matsumoto, “Prediction of Project Cost Overrun Based on
Collaborative Filtering,” Technical report of IEICE. SS, vol.105,
no.229, pp.35-40, 2005.
[9] N. Nagappan, and T. Ball, “Static analysis tools as early indicators of
pre-release defect density,” Proc. International Conference on
Software Engineering (ICSE 05). pp.580-586, 2005.
[10] J. Procaccino, J. Verner, S. Overmyer, and M. Darter, “Case study:
factors for early prediction of software development success,”
Information and Software Technology, vol.44, no.1, pp.53-62, 2002.
[11] Project Management Institute, A Guide to the Project Management
Body of Knowledge: (Pmbok Guide). p.459, Project Management
Institute, 2008.
[12] R Development Core Team, R: A language and environment for
statistical computing. R Foundation for Statistical Computing, 2009.
http://www.R-project.org.
[13] M. Shepperd, and C. Schofield, “Estimating software project effort
using analogies,” IEEE Transactions on Software Engineering, vol.23,
no.12, pp.736-743, 1997.
[14] G. Taguchi, R. Jugulum, The Mahalanobis-Taguchi Strategy: A
Pattern Technology System. Wiley, 2002.
[15] Y. Takagi, O. Mizuno, and T. Kikuno, “An Empirical Approach toCharacterizing Risky Software Projects Based on Logistic Regression
Analysis,” Empirical Software Engineering, vol.10, no.4, pp.495-515,
2005.
[16] J. Verner, W. Evanco, and N. Cerpa, “State of the practice: An
exploratory analysis of schedule estimation and software projectsuccess prediction,” Information and Software Technology, vol.49,
no.2, pp.181-193, 2007.
[1]