Characterization of Risky Projects
based on Project Managers’ Evaluation
Osamu Mizuno
Tohru Kikuno
Osaka University, JAPANYasunari Takagi
Keishi Sakamoto
OMRON Corporation, JAPAN1

June 9, 2000

ICSE2000 • Limerick • Ireland

Outline Background and objective
 Definition of “risky” project Our approach Design of a questionnaire (Step 1)
 Construction of a statistical model (Step 3)
 Risk prediction (Step 5)2June 9, 2000

ICSE2000 • Limerick • Ireland

Background In the company, as a result of process

improvement for 8 years, the number of“confused” projects has decreased.
– However, several projects fall into dangerous statein their development (we call them “risky projects”),
and a few of them rarely cause confusion.
 We recognized such confusion by quantitative

measure after the project finishes.
 However, prediction of the risky projects should

be carried out at the early stage of the project.
3

June 9, 2000

ICSE2000 • Limerick • IrelandObjective
 Our objective is as follows:(1) Identify characteristics of a risky projectbased on the questionnaire and logistic
regression model.
(2) Propose a framework to predict the riskyproject.
(3) Apply the proposed model to actual
projects.
4

June 9, 2000

ICSE2000 • Limerick • Ireland

Outline Background and objective
 Definition of “risky” project Our approach Design of a questionnaire (Step 1)
 Construction of a statistical model (Step 3)
 Risk prediction (Step 5)5June 9, 2000

ICSE2000 • Limerick • Ireland

Risky Project – What’s risky?
 “Risky project” is a confusion-prone project.
– In the company, the SEPG noticed that several

projects tend to be in somehow uncontrollable state
for a certain period of their development.
(Risky project)– Most of them return to controllable state.
– But some rarely fall into a really dangerous state.
(Confused project)
All projects

Risky projects
Confusedprojects

6

June 9, 2000

ICSE2000 • Limerick • Ireland

Evaluation of Risky Projects
 Risky project is evaluated by the SEPG at theend of the development using the quantitativemetrics.
– Using the errors of estimation for cost and duration.
– If actual cost and duration exceed estimated ones

by a certain amount, the project is determined risky.
– All of the projects in this company can be classified

as risky or not.
7

June 9, 2000

ICSE2000 • Limerick • Ireland

Outline Background and objective
 Definition of “risky” project Our approach Design of a questionnaire (Step 1)
 Construction of a statistical model (Step 3)
 Risk prediction (Step 5)8June 9, 2000

ICSE2000 • Limerick • Ireland

Characterization of Risky Projects
Phase 1: We design a questionnaire andconstruct a logistic model to characterize therisky projects.
– Questionnaire:
Extract the intuition from project managers.
– Logistic model:

Analyze the responses objectively.

Phase 2: Using the constructed model, we try topredict the result of new projects.
9June 9, 2000

ICSE2000 • Limerick • Ireland

Outline of Phase 1
Step 1: Design of a questionnaire
Questionnaire
Step 2: Fill in the questionnaire
Responses from managers

Step 3: Construct a logistic model

Phase 2
10SEPG & Osaka univ.
June 9, 2000

Project managers
ICSE2000 • Limerick • Ireland

Outline of Phase 2
Phase 1
Constructed model at Phase 1- 5. 25+ 2. 72 x + 3. 98 x12eE (Y ¦ x1 , x2 ) =- 5. 25+ 2. 72 x1 + 3. 98 x2+1 e

Step 4: Fill in the questionnaire

Responses from managers

Step 5: Calculate the probability

SEPG & Osaka univ.
11

June 9, 2000

Project managersof new projects
ICSE2000 • Limerick • Ireland

Outline Background and objective
 Definition of “risky” project Our approach Design of a questionnaire (Step 1)
 Construction of a statistical model (Step 3)
 Risk prediction (Step 5)12June 9, 2000

ICSE2000 • Limerick • Ireland

Design of Questionnaire (Step 1)
 The questionnaire is designed using the risk

management manuals in the company.
– Thus, it includes long-term experience of thesoftware development.
 Characteristics of the questionnaire:– Five viewpoints for risk factors are included*.
– It can be filled in by (at least) the end of the design.

* Similar views are proposed by SRE model at SEI.

13

June 9, 2000

ICSE2000 • Limerick • IrelandFive Viewpoints for Risk Factors


Requirements–



Estimation–Planning or scheduling method and the resulted project plan.

Project management activity–

14

Staffing of the project and the fundamental skills of developers.

Planning capability–Estimations and technical methods for carrying out theestimation.

Team organization–

Requirement definitions and translation of them intospecification.

Project management activities during the development.
June 9, 2000

ICSE2000 • Limerick • Ireland

Questionnaire (Overview)

15

June 9, 2000

ICSE2000 • Limerick • Ireland

Detail of Questionnaire
2. Estimation
2.1 There were missing items to be estimated; these items
were included in the implicit requirements.
2.2 The importance of estimations was not well recognized.
2.3 Non-technical pressure rendered estimates of costs and/orschedules unrealistic.
2.4 Over-optimism in estimating technical issues.
2.5 Insufficient estimations were carried out using the resultsof successful projects in the past.

For each item, one of the four answers “Stronglyagree”, “Agree”, “Disagree” or “Neither agree nordisagree”, should be returned.
16

June 9, 2000

ICSE2000 • Limerick • Ireland

Outline Background and objective
 Definition of “risky” project Our approach Design of a questionnaire (Step 1)
 Construction of a statistical model (Step 3)
 Risk prediction (Step 5)17June 9, 2000

ICSE2000 • Limerick • Ireland

Construction of the model (Step 3)
 Target projects: 32 projects in 1996 and 1997.
– These projects were done in similar organizations inthe company.
– We delivered the questionnaire to the project

managers of these projects.
– Based on the responses, we estimate thecoefficients in the model.

18

June 9, 2000

ICSE2000 • Limerick • Ireland

Evaluation of Projects by SEPG Since the projects in 1996

and 1997 finished theirdevelopment, the SEPG has
already evaluated whether
they are risky or not.
– Risky: 10– No problem: 22Projects in'96 and '97

Evaluation
by SEPG

PJ1

Risky

PJ2

Risky

PJ3

Risky

PJ4

Risky

PJ5

Risky

PJ6

Risky

PJ7

Risky

PJ8

Risky

....

....

PJ31

No problem

PJ32

No problem

 Note that these 32 projects are19

not all the projects in thecompany.
June 9, 2000

ICSE2000 • Limerick • Ireland

Conversion of Source Data
 For each answer to the item, we assign thefollowing points:
Answer–20

Point

Strongly Agree

2Agree1Disagree0

Neither agree nor disagree

1We assigned point 1 to “Neither agree nor disagree” because
the fact the manager cannot say “disagree” implies someproblem in the project.

June 9, 2000

ICSE2000 • Limerick • Ireland

Conversion of Source Data (cont’d) We summed up and normalized these points

for each viewpoint (risk factor).
2. Estimation

21

Evaluation

2.1

There were missing items to be
estimated; these items were included in
the implicit requirements.

1
Agree

2.2

The importance of estimations was notwell recognized.

0Disagree

2.3

Non-technical pressure renderedestimates of costs and/or schedules
unrealistic.

Neither agree1
nor disagree0.82.4

Over-optimism in estimating technicalissues.

0Disagree

2.5

Insufficient estimations were carried out
using the results of successful projects
in the past.

Strongly
2agreeJune 9, 2000

ICSE2000 • Limerick • IrelandSource Data from Projects (’96 & ’97)

Projects in'96 and '97

Requirements

Estimation

Team
organization

Planning
capability

Projectmanagement
activity

Evaluation
by SEPG

PJ1

1.50

1.00

0.50

0.83

1.00

Risky

PJ2

1.50

1.00

0.50

1.17

1.00

Risky

PJ3

0.75

1.40

1.00

0.50

0.17

Risky

PJ4

1.50

0.40

1.00

0.83

0.83

Risky

PJ5

1.00

1.00

0.50

0.50

0.83

Risky

PJ6

0.75

0.00

0.00

1.00

0.33

Risky

PJ7

1.50

1.40

2.00

1.67

1.17

Risky

PJ8

1.75

1.80

2.00

1.83

1.33

Risky

....

22

....

....

....

....

....

....

PJ31

0.00

0.00

0.00

0.17

0.00

No problem

PJ32

0.00

1.20

0.50

0.17

0.33

No problem

June 9, 2000

ICSE2000 • Limerick • Ireland

Logistic Regression Modelb0 + b1 x1 + b2 x2 +... + bn xne
E (Y ¦ x1 , x2 , ... , xn ) =b0 + b1 x1 + b2 x2 +...+ bn xn1+ e
 Objective variable Y: Risky(1) or not(0).
 Explanatory variables x1, ..., xn: Risk factors.
– E(Y|x1,...,xn) denotes the conditional probability thata project becomes risky.

 We must estimate stepwisely the values ofcoefficients (b0,...,bn) using the source data.
23

June 9, 2000

ICSE2000 • Limerick • Ireland

Prediction Model

Two risk factors “Estimation” and “Planning Capability”
are adopted in the model as x1 and x2, respectively.
- 5. 25 + 2. 72 x1 + 3. 98 x2eE (Y ¦ x1 , x2 ) =- 5. 25 + 2. 72 x1 + 3. 98 x2
1+ e
 Result of application to the source projects.

Predicted
Actual24

No problem

Risky

No problem

184Risky

1

9June 9, 2000

27 projects out of 32
were predicted
correctly.
(The threshold between risky or
not is selected as 0.3)

ICSE2000 • Limerick • Ireland

Outline Background and objective
 Definition of “risky” project Our approach Design of a questionnaire (Step 1)
 Construction of a statistical model (Step 3)
 Risk prediction (Step 5)25

June 9, 2000

ICSE2000 • Limerick • Ireland

Prediction of Risky Projects (Step 5)
 By applying the prediction model E(Y|x1,x2),

we tried to predict the result of 8 projects in1998.

– These projects also have almost the same property

as the projects used in construction of the model.
– Since the development of these projects havealready finished, we can compare the result ofprediction with the result of actual evaluation by theSEPG.
26

June 9, 2000

ICSE2000 • Limerick • Ireland

Outline of Phase 2
Phase 1
Constructed modelfrom 1996 & 1997 at Phase 1- 5. 25+ 2. 72 x + 3. 98 x12e
E (Y ¦ x1 , x2 ) =- 5. 25+ 2. 72 x1 + 3. 98 x2+1 e

Step 4: Fill in the questionnaire

Responses from managers

Step 5: Calculate the probability

SEPG & Osaka univ.
27

June 9, 2000

Project managers
of projects in 1998.
ICSE2000 • Limerick • Ireland

Result of Experiment (’98)

0.99Actual
Evaluation
by SEPG
Risky

1.33

0.97

Risky

1.20

0.33

0.34

Risky

PJ36

0.00

0.17

0.01

No problem

PJ37

0.40

0.33

0.06

No problem

PJ38

0.20

0.33

0.03

No problem

PJ39

0.00

0.17

0.01

No problem

PJ400.40

0.67

0.18

No problem

PJ33

0.60

Planning
Capability
(x2)
2.00

PJ34

1.20

PJ35

Projects in’98

Estimation(x1)

E(Y|x1,x2)

PJ33, 34, 35 are considered to be risky.

(Again, the threshold of risky or not is assumed as 0.3.)
28

Results of actual evaluation are the same as theresults of prediction by the model.
June 9, 2000

ICSE2000 • Limerick • Ireland

Conclusion
 We proposed a new approach to predict therisky projects based on the questionnaire andstatistical model.
– Experimental evaluation shows that the proposedapproach has high predicting capability.

Future Work– Apply to the ongoing projects.
– Improve the design of questionnaire.
– How to mitigate the risk.

29June 9, 2000

ICSE2000 • Limerick • Ireland