Estimation of Project Success Using Bayesian Classifier
Seiya Abe, Osamu Mizuno, Tohru Kikuno

Nahomi Kikuchi, Masayuki Hirayama

Graduate School of Information Science andTechnology
Osaka University
1-5 Yamadaoka, Suita, Osaka, 565-0871, Japan

Software Engineering Center
Information-technology Promotion Agency
2-28-8 Honkomagome, Bunkyo-Ku, Tokyo,113-6591, Japan{s-abe, o-mizuno, kikuno}@ist.osaka-u.ac.jp{n-kiku, m-hiraya}@ipa.go.jp

ABSTRACTThe software projects are considered to be successful if thecost and the duration are within the estimated ones andthe quality is satisfactory. To attain project success, theproject management, in which the ﬁnal status of project isestimated, must be incorporated.
In this paper, we consider estimation of the ﬁnal status(that is, successful or unsuccessful) of project by applying
Bayesian classiﬁer to metrics data collected from project. In
order to attain high estimation accuracy rate, we must select only a set of appropriate metrics to be applied. Herewe consider two selection methods: the ﬁrst method by theexperts and the second method by the statistical test.
Then we conducted an experiment using 28 project data
and 29 metrics data in an organization of a certain company. The result showed that the method by the test gavehigher accuracy rates than the method by the experts, andBayesian classiﬁer with the test method is eﬀective to estimate project success.
Categories and Subject Descriptors: K.6.3 [Software
Management]: Software development
General Terms: Measurement
Keywords: project success, estimation, Bayesian classiﬁer

are rather limited. Thus the prediction of project success atearly stage is one of the important tasks for project management.
In this paper, we try to predict project success at theearly stage of the software development. If a project is estimated to be unsuccessful, then appropriate treatment canbe applied to change the project into successful one.
Here we apply the Bayesian classiﬁer to metrics data collected from development process in order to predict projectsuccess. By the result of the preliminary experiments, thecapability of the prediction was veriﬁed within limit, but
the accuracy rate of prediction was not satisfactory. Thusin this paper, we try to select an appropriate set of metrics
in order to increase the accuracy rate of the prediction.
We propose two selection methods and compare the accuracy rates of two methods. The ﬁrst method is to selectmetrics by the experts. Thus the key idea is exactly theknow-how or the experiences by the experts. On the other
hand, the second method applies statistical test to project
data in order to select the metrics. Then we evaluate theeﬀectiveness of two methods by applying the Bayesian classiﬁer with two methods to 28 practical project data in an
organization of a certain company.

2. PRELIMINARIES
1.

INTRODUCTION

From the standpoint of industries, it is deeply desired tolead all of the software projects into success ones. Generally
speaking, project success is considered from three distinct
viewpoints: the quality of the product, the cost of development and the duration of the project.
Recently it is said that the projects become more andmore diﬃcult to be accomplished successfully. As the main
reasons for this, the increases in the functionality and thecomplexity of the projects are pointed out [2]. Additionallythe demands on the short term development and the small
amount of budget are clearly other practical reasons. In
fact, many studies reported several unsuccessful projects insoftware development [1, 4].
On the other hands, there are many innovative researches
which try to lead projects into success [3, 5–7]. The mostfundamental way is an improvement in the test activity or
the review activity of the development process. It is easy toincorporate into the project, but unfortunately the eﬀects
Copyright is held by the author/owner.
ICSE’06, May 20–28, 2006, Shanghai, China.
ACM 1-59593-085-X/06/0005.

2.1 Project Context
Software development projects must be performed by developers under the constraint of limited resources. They
must be executed by balancing the competing demands for
quality, time, and cost. In order to terminate the projects
successfully, a project plan must be prepared and the projects
must be controlled by the managers according to the plan.
In this research, we deal with 28 software development
projects in an organization of a certain company. Here
we describe project context of these 28 projects as follows:(1) They were the development and integration of softwaresystem for speciﬁc corporate customers. (2) As the ﬁnal
product, high reliability system is strongly needed. (3) The
development was performed in the same development processes. (4) The size of all projects are middle or large. (5)The company is responsible for all these projects.

2.2 Software Metrics
Software measurement and quantiﬁcation are crucial toenable control of projects or processes. Software metrics
provide a quantitative basis for the development. Metrics
are indicators which reﬂect the situation of process execu-

tion. Since the result of process execution inﬂuences on thequality of products, the cost, and the period of the development projects, it is thought that metrics are related to thesethree aspects.
In this research, we consider 29 metrics shown in Table 1
that are classiﬁed into ﬁve categories: development process,project management, company organization, human factor
and external factor. These metrics are applied to the 28
projects and the data are collected. Each metric is evaluated by ordered scale with 5 degrees. We cannot present
the detailed deﬁnitions of these metrics according to thecontract with the company.
Table 1: All metrics
1 Metrics on Development Process
M1: Familiarity of technology used
M2: Necessity of adjustment to the other parties
M3: Requirements elaboration
M4: Pending requirements
M5: Potential requirements change
M6: Complexity of system features
M7: Difficulty of integration of COTS
M8: Performance and HW constraints
M9: Feasibility of performance requirements
M10: Experience of platform
M11: Stability of platform
M12: Amount of documentation
M13: Design consideration of reliability, safety, security
M14: Risks of tools used
2 Metrics on Project Management
M15: Planning of the whole project
M16: Planning of test environment
M17: Size and cost estimation
M18: Availability of system and support
M19: Communication among project stakeholders
3 Metrics on Company Organization
M20: WBS
M21: New customer
M22: Customer’s participation to requirements definition
M23: Customer’s participation to acceptance test
M24: Existence of process manuals and adherent execution of process
4 Metrics on Human Factor
M25: Skill and experience of developers
M26: Skill and experience of PM
M27: Skill and experience of designers
M28: Continuity of key members
5 Metric on External Factor
M29: Requirement volatility

2.3 Quality, Cost and Duration
It is strongly required that all projects deliver the required
products on the scheduled time within the estimated budget. Thus we should evaluate project success from three
viewpoints: the quality of product, the cost of development
and the duration of project. The followings show the criteria
to judge successfulness of a given project.
Quality: The quality is evaluated by the number of defects found during acceptance test by user. If the number isless than #1 (a threshold), then the project is evaluated tobe successful. Here we set #1 to be 0.
Cost: Since the cost is not available for us, we adopt thevalue of development eﬀort instead of the cost itself. If thediﬀerence between the actual eﬀort and the estimated one
is less than #2(a threshold), the project is evaluated to be
successful. Here we set #2 to be 10 percentage.
Duration: The duration is evaluated by the development
period. If the diﬀerence between the actual period and theestimated one is less than #3(a threshold), the project isevaluated to be successful. Here we set #3 to be 1 day.
For the threshold #1, #2, #3, we determined actual values such as 0, 10 percentage, and 1 day according to discussions with managers in the company.
Table 2 shows the judgement of 28 projects. The judge-

ment was done according to the criteria mentioned above.
In the table, ”-” indicates that the judgement was not done
for the viewpoint since necessary data is not available.
Table 2: Judgement of project success
Project ID
P1
P2
P3
:
P28

Viewpoints
Quality
Unsuccessful
Successful

Cost
Successful
Unsuccessful
Successful

Duration
Successful
Successful
Successful

:

:

:

Successful

Successful-

2.4 Research Objective
The ﬁnal goal of this research is to predict the ﬁnal status(that is successful or unsuccessful) of a project using metrics data. As we mentioned in subsection 2.3, there are three
key viewpoints to evaluate project success: quality of product, cost of development and duration of project. Thus weconsider the status of the project from these three viewpoints.
In this paper, we apply Bayesian classiﬁer to metrics data
to predict the ﬁnal status of the project. Then, as we showedin subsection 2.2, we can use 29 metrics for each project.
By observing the metrics carefully, we notice that somemetrics are strongly related to a certain viewpoint, but are
not eﬀective to others. It brings us an important research
objective that how to select useful metrics in order to predictthe ﬁnal status with respect to three viewpoints.

3. BAYESIAN CLASSIFIER
The naive Bayesian classiﬁer is one of the most common
approach to classify categorical data into several classes. Let
Q1 , Q2 , · · · , Qn be the variables denoting the observed attribute values to predict a discrete class C. Furthermore, let
c represent a particular class. Given the values q1 , q2 , · · · , qn ,we can use Bayes’ theorem to calculate the probability P (C =c|Q1 = q1 ∧ · · · ∧ Qn = qn ) and then predicts most probableclass. By the Bayes’ theorem, this probability is expressed
as follows:ni=1 P (Qi = qi |C = c)
× P (C = c).
P (Q1 = q1 ∧ · · · ∧ Qn = qn )The probability P (C = c) can be easily estimated fromtraining data. In this research, we deal with numeric attributes. Each numeric attribute is modeled by some continuous probability distribution over the range of its values.
We assume that within each class, the values of numeric attributes are normally distributed. Generally P (Q1 = q1 ∧· · · ∧ Qn = qn ) is not estimated as it is just a normalizing
factor. Instead, we can ignore it and then normalize so thatthe sum of P (C = c|Q1 = q1 ∧ · · · ∧ Qn = qn ) over all classes
is 1. Thus we can compute P (C = c|Q1 = q1 ∧· · ·∧Qn = qn )and predict the value of C if the values of qi are given.
In this paper, we let Qi denote a risk factor or a metric and C denote the status of a project with two values:s(successful) and u(unsuccessful). From the calculated probability, we can classify the test data into two classes. Here,
we determine a project is in class s if P (C = s|Q1 = q1 ∧· · · ∧ Qn = qn ) ≥ P (C = u|Q1 = q1 ∧ · · · ∧ Qn = qn ).

Table 3: Evaluation by expert method
Quality

Cost

Duration

Predicted (# of Projects)
Successful Unsuccessful
Successful
16
4Unsuccessful
3
1
Accuracy rate: 0.708

Predicted (# of Projects)
Successful Unsuccessful
Successful
14
5
Unsuccessful
80Accuracy rate: 0.519

Predicted (# of Projects)
Successful Unsuccessful
Successful
7
7
Unsuccessful
6
6
Accuracy rate: 0.500

ActualActualActualTable 4: Evaluation by test method
Quality
Predicted (# of Projects)
Successful Unsuccessful
Successful
200Unsuccessful
3
1
Accuracy rate: 0.875
Actual

4.

Cost

Duration

Predicted (# of Projects)
Successful Unsuccessful
Successful
14
5
Unsuccessful
3
5
Accuracy rate: 0.704

Predicted (# of Projects)
Successful Unsuccessful
Successful
113Unsuccessful
3
9
Accuracy rate: 0.769

ActualEXPERT METHOD

4.1 Selection by the Experts
In order to ﬁnd a set of metrics from 29 metrics which isused in Bayesian classiﬁer, three experts discussed the eﬀectiveness of each metric with respect to three viewpoints inbrainstorming. All of the experts have a wealth of businessexperiences in software development and understand metrics
quite well.
For each metric, they determine the viewpoints for which
the metric seems to be eﬀective. Please note that somemetrics are determined to be eﬀective for several viewpoints.
As the result, the selection of metrics are done. For thequality of product, 12 metrics, M7, M8, M9, M10, M11,
M12, M13, M19, M24, M25, M27, and M28 are selected tobe eﬀective. For the cost, only 3 metrics, M12, M14, andM29 are selected. Finally for the duration, 18 metrics, M1,
M2, M3, M4, M5, M6, M7, M15, M16, M17, M18, M19,
M20, M21, M22, M23, M25, and M26 are selected. It isobserved that each metric is considered to be eﬀective to atleast one viewpoint.

4.2 Application of Bayesian Classifier
In order to apply Bayesian classiﬁer to predict ﬁnal status of projects with respect to viewpoints, we must prepare
two kinds of data: the data for learning and the data forprediction. In this paper, we adopt the leave-one-out crossvalidation method. Thus we choose one project as the datafor prediction, and predict its status using the data of therest 27 projects. By executing the prediction interchangeably for each project, we get all predicted results.
In case of prediction with respect to the quality of product,
we consider M7, M8, M9, M10, M11, M12, M13, M19, M24,
M25, M27, M28 as Q1 , Q2 , Q3 , Q4 , Q5 , Q6 , Q7 , Q8 , Q9 ,Q10 , Q11 , Q12 , respectively.
The result of quality prediction is shown in Table 3. The
table shows that 19 projects are predicted to be successful. Among them, 16 projects are actually successful and3 projects are unsuccessful. On the other hand, 5 projects
are predicted to be unsuccessful, and 4 projects are actually
successful and 1 project is unsuccessful among them. The
accuracy rate is calculated to be (16 + 1)/24 = 0.708.
In the same way, the cost and the duration are predictedby applying Bayesian classiﬁer, and the evaluated results areshown in Table 3. The accuracy rates for the cost and theduration are 0.519 and 0.500, respectively.

Actual

5. IMPROVEMENT IN ESTIMATION
5.1 Selection by Statistical Test
In order to improve the accuracy rate of prediction, we
propose to perform the Wilcoxon rank sum test on project
data to select more eﬀective metrics out of 29 metrics.
Generally, the Wilcoxon rank sum test can determine if
one population is statistically diﬀerent from another by comparing the locations of two populations. In this paper foreach metric, we apply the test to check if there is a statistical diﬀerence of the distributions between the group of
successful projects and the group of unsuccessful projects.
If such a diﬀerence is observed for a certain metric, then themetric seems to have strong inﬂuence on the prediction.
We adopt the probability value(that is, p-value) to checkif there is a statistical diﬀerence. The smaller the p-valueis, the more statistical diﬀerence is observed between two
groups. However since we need several metrics to be applied to Bayesian classiﬁer, we set p-value to be 0.3(relatively large compared with ordinary values 0.05 or 0.1).
At ﬁrst the Wilcoxon rank sum test is applied to project
data shown in Table 2 with respect to the quality and data
for metrics M1 through M29. As the result, 10 metrics, M1,
M9, M18, M19, M20, M25, M26, M27, M28 and M29 areselected to be eﬀective with respect to the quality. Then,for the cost the Wilcoxon test is applied similarly and 4
metrics, M6, M14, M21 and M28 are obtained. Finally forthe duration the Wilcoxon test is applied and 8 metrics, M7,
M9, M10, M19, M21, M22, M25 and M27 are obtained. It
is clearly observed that 13 metrics are not selected for any
viewpoint by the Wilcoxon test, although all metrics are
selected by the experts in subsection 4.1.

5.2 Application of Bayesian Classifier
In order to predict ﬁnal status of projects, we applied
Bayesian classiﬁer. In the application, we again adopt theleave-one-out cross validation method to project data shownin Table 2. Here we use metrics data which are newly selected by the Wilcoxon rank sum test.
The result of application of Bayesian classiﬁer is shown in
Table 4. For the quality of product, 21 projects are predicted
correctly and only 3 projects are failed to predict. Thus
the accuracy rate is 0.875 and thus it is improved quite wellcompared with 0.708 by the experts. It should be noted thatfor 20 projects, which are actually successful, the Bayesianclassiﬁer can predict correctly.
Next, for the cost of development, the accuracy rate be-Table 5: Comparison of accuracy rate
MethodQuality
Expert method 0.708
Test method
0.875

Cost
0.519
0.704

Duration
0.500
0.769

comes 0.704. In this case, compared with the accuracy rate
0.519 by the experts, it is also improved very much. By
just looking at the resultant tables, we notice that out of 8
projects, which are actually unsuccessful, the Bayesian classiﬁer can predict 5 projects correctly. However for the rest
3 projects the classiﬁer failed to predict.
Finally for the duration of project, the accuracy rate becomes 0.769. In this case, compared with the accuracy rate
0.500 by the experts, it is highly improved.

6.

DISCUSSIONS

Table 5 summarizes the accuracy rates of two methods
and Table 6 shows comparison of metrics selected by two
methods. At ﬁrst, we can see that signiﬁcant improvementsin accuracy rates are attained by the test compared with theone by the experts. The reason for this improvement is thediﬀerence between the selected metrics, since both methods
apply the same Bayesian classiﬁer to the project data.
We start to analyze the characteristics of the comparisonin Table 6. Out of 29 metrics which are selected by the experts, only 16 metrics are chosen by the test. Thus the rest
13 metrics are not selected in the method using the test.
For the viewpoint of the quality, 12 metrics are selected by
the experts and 10 metrics are selected by the test. Comparing 12 metrics and 10 metrics, 5 metrics are commonly
contained. For the viewpoint of the cost, 3 metrics are selected by the experts and 4 metrics are selected by the test.
These two sets of metrics are diﬀerent so much, since only
one metric M14 is contained commonly. For the viewpointof the duration, 18 metrics are selected by the experts andonly 8 metrics are selected by the test. Out of 18 metrics,
5 metrics are commonly selected by the test, and 8 metrics
are completely removed.
Firstly for the quality, Table 5 shows that relatively highaccuracy rates are attained by both methods and that a further improvement is accomplished by the method with test.
As mentioned earlier in subsection 5.2, the improvement isrealized by the correct prediction of successful projects. On
the other hand, for unsuccessful projects the accuracy rateis still not satisfactory. The main reason is that the numberof projects, which are actually unsuccessful, is only 4 andthus very small compared with 20. Thus it seems to be very
diﬃcult to predict unsuccessful projects correctly.
Next for the cost, Table 5 shows that the accuracy rate isimproved very much. In this case the improvement is realized by the increase of the accuracy rate of prediction withrespect to unsuccessful projects. That is, all unsuccessfulprojects are predicted incorrectly in Table 3, but 5 out of 8
are predicted correctly in Table 4. This essential change inthe prediction may be incurred by the changes in selected
metrics as shown in Table 6.
Finally for the duration, Table 5 shows that the accuracy
rate is highly improved. It is realized by the increase of thenumber of the correct predictions for both successful andunsuccessful projects. As pointed out before, the numberof metrics in subsection 4.1 is too large compared with theTable 6: Comparison of selected metrics
Metrics Expert method
Test method
M1 Duration
Quality
M2 Duration
M3 Duration
M4 Duration
M5 Duration
M6 Duration
Cost
M7 Quality, DurationDuration
M8 Quality
M9 Quality
Quality, Duration
M10 Quality
Duration
M11 Quality
M12 Quality, Cost
M13 Quality
M14 Cost
Cost
M15 Duration

Metrics Expert method
Test method
M16 Duration
M17 Duration
M18 Duration
Quality
M19 Quality, Duration Quality, Duration
M20 Duration
Quality
M21 Duration
Cost, Duration
M22 Duration
Duration
M23 Duration
M24 Quality
M25 Quality, Duration Quality, Duration
M26 Duration
Quality
M27 Quality
Quality, Duration
M28 Quality
Quality, Cost
M29 Cost
Qualitynumber of project data. Thus some metrics are generally
related to the duration, but judging from the project data
they are not so associated with it. Therefore the number of
metrics is reduced by the test.

7. CONCLUSION
In this paper, we have predicted the ﬁnal status(that is,
successful or not) of a given project by applying the Bayesianclassiﬁer to metrics data collected from the project. The
ﬁnal status is evaluated from the three distinct viewpoints:
the quality of the product, the cost of development and theduration of the project.
In order to attain high accuracy rate of the prediction,
selecting an appropriate set of metrics is a key step. Here
we present two methods: the selection by experts and theselection by the statistical test. The experts select metrics
using so-called know-how, and the test can select metrics byapplying statistical knowledge. The experiments using an
actual data in an organization of a certain company showed
the superiority of the test over the experts.
The future research works include further analysis of theresult in this paper, and evaluation of Bayesian classiﬁer
under another project context.

8. ACKNOWLEDGMENTS
Authors would like to thank Oki Electric Industry Co.,
Ltd. for their cooperation.

9. REFERENCES[1] R. L. Glass. How not to prepare for a consulting assignmentand other ugly consultancy truths. Communication of theACM, 41(12):11–13, 1998.
[2] R. L. Glass. Software runaway - some surprising ﬁndings.
Journal of Systems and Software, 41(2):75–77, 1998.
[3] J. Jiang and G. Klein. Software development risks to projecteﬀectiveness. Journal of Systems and Software, 52:3–10, 2000.
[4] S. McConnell. Rapid Development. Microsoft Press, 1996.
[5] J. D. Procaccino, J. M. Verner, S. P. Overmyer, and M. E.
Darter. Case study: factors for early prediction of softwaredevelopment success. Information and Software Technology,
44:53–62, 2002.
[6] Y. Takagi, O. Mizuno, and T. Kikuno. An empirical approach
to characterizing risky software projects based on logistic
regression analysis. Empirical Software Engineering,10(4):495–515, 2005.
[7] C. Wohlin and A. A. Andrews. Prioritizing and assessing
software project success factors and project characteristics
using subjective data. Empirical Software Engineering,
8:285–303, 2003.