Predicting Runaway Projects for Reliable Software Developments
Using Bayesian Classifier
Seiya ABE, Osamu MIZUNO, and Tohru KIKUNO
Graduate School of Information Science and Technology, Osaka Universitys-abe, o-mizuno, kikuno  @ist.osaka-u.ac.jp
GIH JIKML

1. Introduction
In order to realize reliable software development, problems in software development must be detected and avoided
as soon as possible. Thus, detecting signs of problems at
an early stage of the software project is important. Much
research has been carried out on the detection of problem
signs of software development projects [2, 3]. This study
shows an easy-to-use approach to predict runaway projects
in an organization to achieve reliable development process.
Experimental results of predicting runaway projects are alsoshown.

2. Predicting Runaway Project by BayesianClassifier
Figure 1 shows the outline of our approach to make a
Bayesian classifier to predict runaway projects. First, inStep 1, we designed a questionnaire to be distributed toproject members to collect the assessment data. The questionnaire is constructed based on our previous works andvarious software risk research [4]. Next, in Step 2, theSEPG distributed the questionnaire to project members, andasked them to fill out the questionnaire. After they finished filling out the questionnaire, the SEPG collected responses to the questionnaire. At the same time, in Step 3,
the SEPG determined the success variable and the value of
it for each development project from available quantitative
project data. Finally, in Step 4, we apply the naive Bayesianlearning to the responses to the questionnaire, which areprepared for the model learning. In case of prediction,
we apply the learned model to responses of new projects
and the value of success variable is determined by obtainedprobability.
The naive Bayesian classification is an optimal methodof supervised learning if the values of the attributes of an
example are independent given the class of the example.
Although this assumption is almost always violated in practice, recent research has shown that naive Bayesian learning
is also effective in practice [1].

GIH JIKP"!   # $  % $'& $( ) *+     *" #, - . / 01- 2 - 3 45@>6 - 57 8 84 03 22 3 80 ?3 4 109 ?9 3:8 83 ;0 A 8 8 < 4 00 ?7 0 4 3 =
6 1? B- : 4 2 8 3 8 9 - 2 - 3 4CD  'E$ %F !  # $ CD# $ %!  TU & $ V  \ W # p^ g _ f ` b d a b j c _f b dq e _ e f gr _ j d q c k j e b s j f b q e [uvhh ci j q et _ e l b ef kd lj j e m j j k n _ b d i b f o_ e f ]^xz{| {}

GQH JIKXCD #  * # $ E$ %!  TUE& $ V W # YD &$  Z# $ % F !   # $  N ) ) O$ ! #F !  # $ ^ _ ` a bc_ d _ e fg][ \\rp _ g j f b dd j q f cb qk ej e b s j f b q e ]hh ic j q et _ e l b ef kd lj j e m j j k n _ b d i b f o_ e f

GQH JQKSR     CD# $ % ! W W  E& $ V  \\ W # \^ _ ` a bc _ d _ e fgp g fbd j fbq e\rh _ i j j e d e b qe ck k jl je mb s j j n f bb qi b f e o ]h cq t_ l f d j e j k _ d _ e fwxyypww        

Figure 1. Bayesian classification

3. Design of the Questionnaire
The questionnaire consists of 30 questions to ask about
possible signs of problems in software development. Eachquestion must be filled in the specified ranks. For constructing this questionnaire, we first quoted the questionnaire used in our previous study [4]. We then discussed with
the SEPG members in the company to be applied this questionnaire and determine company-specific questions. (SeeTable 1.)
Table 1. Overview of questionnaire° ± ­® «¨ ¬ ©ª   ~    ~ ¢ ¥   ¦§ ~ ¢ ¥   ~¤ ¤¢  £  ~ ¡   E ~   + ~  E ~ '       ~ 
~ ¯®©ª § 

4. Evaluation with Empirical Data
4.1. Target Projects and Success Variable
The questionnaire was delivered to total 116 project
members in 19 development projects in a certain company
on July, 2003. The responses were collected one monthTable 2. Responses of questionnaire (a part)²U³'´µ¶·D´'³¹¸ º ºU³»³¼ ¶µD³½§¸ º ¾½ ¶¿ ³'ÀÁ§¸ ºÃÂÄMÂÅÆÂOÇÈÂOÉÃÂOÊÃÂËÈÂOÌÈÂÍÈÂÎMÂÄÏÑÐ Ð ÐÒÂOÇÏÓÔÕÔÖUÔ×Ø×ÚÙÛ×ÜÔ×ÝÔÞÔß ß ßàÔÓ§×Õ§×ÖD×ÔØÔØÔØÔÛÔØÔá×ÜÔØÔÞÔß ß ßâ×ÓUÙÕUÙÖD×ÔØÔá×Ø×ÜÔá×Ø×ÜÔá×ãÔß ß ßàÔÓäÕäÖD×ÙÛ×ÚÙåäØ××ÜÔá×ãÔßßßÓUæÕUæÖD××ÝÔØÔÔá×ßßßÓUçÕUçÖD××ÝÔèÙÛ×Û×ÚÙÛ×ÜÔá×ß ß ßéÙÓUêÕUêÖD×ÙÛ×ÚÙåäÚÙØÙØÙëÔØÔìæß ß ßàÔÓUíÕUÙÖÙÔØÔá×Ø×Û×Ø×Ø×Û×Ø×ãÔß ß ßâ×ÓUîÕUæÖÙÔØÔØÔÔßßßÓEÔ ïÕUçÖÙÙÛ×ÝÔá×Û×ÚÙÛ×Û×ÝÔìÙß ß ßâ×ÓEÔÔÕUêÖÙÙÜÔØÔØÔáÙØÙåäØ×ÝÔìæß ß ßàÔÓEÔ×ÕUíÖäÔØÔØÔØÔÛÔØÔá×Û×Ø×ãÔß ß ßàÔÓEÔ ÙÕUÙÖäÔØÔá×Ø×Û×Ø×Ø×Û×Ø×ãÔß ß ßâ×ÓEÔ äÕäÖä×Ø×Ø×Ø×Û×Ø×Ø×ÜÔá×ßßßÓEÔ æÕUæÖäÔØÔØÔÔØÔßßßÓEÔ çÕUçÖäÔØÔá×ÝÔÛÔØÔØÔð×Ø×ãÔß ß ßàÔÓEÔ êÕUêÖä×ÝÔØÔá×ØÙØÙÛ×ÜÔØÔìæß ß ßàÔÓEÔ íÕUíÖæäØäØäØäÛäØäëÙèäØäñæß ß ßòäÓ§íïÕUêóóØóÛóØóØóØóÛóØóØóÞóÖUÔæôÙÛ×Ø×Ø×ØÙØÙØÙå×Ø×õæßßßÔ

ó

later. These 19 projects were development of software products related to the software process improvement activities
in this company. Among them, 15 projects had already finished their development in 2001 and 2002, and the rest 4projects were on-going projects in 2003.
In this experiment, since SEPG has to determine thevalue of success variable, the projects must be finished. For
this reason, we used 80 responses in 15 projects in 2001 and2002. A part of collected 80 responses is shown in Table
2. We can see that some of responses have blank answers.
Since Bayesian classification allows incomplete data suchthat some of questions are left no answer, we can use thecollected data as they are.
In order to evaluate the final status of software development projects, we define a subjective metric STATUS which
represents whether a project is runaway or success. To determine the value of STATUS, several aspects of projects
such as cost, quality, and duration are considered. It is evaluated by SEPG members without referring the responses ofthe questionnaire but considering quantitative data obtained
from these projects. The result of evaluation for 15 projects
are shown in “Actual Result” column of Table 3.

4.2. Evaluation
In this experiment, we tried to show prediction of STATUS for a project from the project member’s response byapplying naive Bayesian classifier. In order to show theeffectiveness of the proposed approach, we performed thejackknife method on the collected data. Simply speaking,we first excluded responses related to a certain project for
testing. Next, we consider the other responses as learning data. Finally, we applied the excluded responses tothe Bayesian classifier and obtained predicted STATUS for
applied responses. This procedure was iterated until all
projects were predicted.
Project members in a project have their own opinion of
the problem signs in their project. There could exist different responses to the questionnaire for a project. Thus,predicted results from responses for a project could vary.
We therefore determine whether a project is runaway or notwith a majority voting of predicted results.
Table 3 shows predicted STATUS for 80 responses, andTable 3 also shows classified results of prediction for eachproject. Each row shows a project ID, its actual STATUS ofdevelopment (Success or Runaway), and result of prediction by the Bayesian classifier using responses from project
members. Grayed rows indicate that the prediction of a
project is the same as the actual result by majority voting.
For example, project P15 has 6 members who responded
the questionnaire. The result of prediction based on the responses of 4 members out of 6 shows that P15 is to be success. By majority voting, P15 can be considered to be success, and the predicted result is correct since actual result ofP15 was success. We can see that 12 out of 15 projects werecorrectly predicted.
At this point, we cannot state whether or not the accuracyof prediction is good enough. To do so, we need to perform
more experiments. This is our important future research.
Table 3. Comparison of actual STATUS and predicted result  %$    ÷"ú ö ÷  "
÷  #  "  !
!
öø÷"ù"úûüúEý
 "ÿ
  
    ö ÷     ÷        Eùöø÷"ùEú'û ú"ý


ÿ
ÿ

ÿ



þ




ÿ









ÿ

ÿ
ÿ



ÿ

ÿ
þ

Acknowledgment Authors would like to thank Mr. Kenji
Matsuse and Mr. Hidenobu Sakata of RICOH Company,Ltd. who gave us empirical data of their software development and invaluable advise to this research.

References[1] P. Domingos and M. J. Pazzani. On the optimality of the simple Bayesian classifier under zero-one loss. Machine Learning, 29(2-3):103–130, 1997.
[2] J. Jiang and G. Klein. Software development risks to project
effectiveness. Journal of Systems and Software, 52:3–10,
2000.
[3] J. D. Procaccino, J. M. Verner, S. P. Overmyer, and M. E.
Darter. Case study: factors for early prediction of softwaredevelopment success. Information and Software Technology,
44:53–62, 2002.
[4] Y. Takagi, O. Mizuno, and T. Kikuno. An empirical approach
to characterizing risky software projects based on logistic regression analysis. Empirical Software Engineering, (to appear).