M O D E R N I Z AT I O N

CLEARING A PATHWAY TO SUCCESSTA B L E O F C O N T E N T S
INTRODUCTION......................................................................... 2C A S E C O M P A R I S O N. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
C A S E 1 : N E W A P P L I C A T I O N D E V E L O P M E N T.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
C A S E 2: C O M M E R C I A L A P P L I C AT I O N PA C K A G E . ..................... 8C A S E 3: M O D E R N I Z AT I O N ....................................................... 10
PROOF POINTS. ........................................................................ 12S U M M A R Y.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 5
A B O U T T H E V I R T U A L A D V I S O R ®. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 6

INTRODUCTION
In a recent engagement we evaluated the order processing application of one of our clients. This organization takes
orders around the world, 24/7. Orders come in from the company’s website, call centers, faxes, and bulk e-mails.
Much of the process is manual, and inventory is constantly out of balance. The new proposed order processing
application would automate a great deal of the processing and improve inventory control. It is estimated that thecurrent system is costing the company $5,000 a day in lost orders, extra labor, and additional inventory. The sooner
the new system is deployed, the sooner the firm will reap savings.
The current system has hampered the organization’s ability to increase its competitive advantage, extended time
to market for new, innovative products, and increased the risk of falling behind competitors as well as not meetingcompliance regulations.
Our client is currently using an IBM z10 mainframe system for running its order processing application. The
company has found it too expensive and difficult to modify and maintain the application. It is also one of the last
applications running on the IBM mainframe. The application consists of two million lines of COBOL/CICS code.
The company plans to upgrade the application and reduce total cost of ownership by running it on commodity
hardware and software.
The company has three choices for upgrading the order processing application: rewrite the application fromscratch using new tools and techniques, buy a package off-the-shelf, or modernize the current application. The
organization solicited bids to upgrade the application, which we reviewed. There were three bids for each typeof upgrade. The average bid for new development was $10 million for a three-year project. The average bid fora package was $5 million and a two-year implementation project. The average bid for a modernization project
was $3.5 million and just a one-and-a-half-year project. Because of the lucrative payback all three methods had apositive return.

2

M O D E R N I Z AT I O N C L E A R I N G A PAT H WAY T O S U C C E S S

INTRODUCTIONT

he organization was concerned with both long-term and short-term gain and reduced risk, both forbusiness and technology. Cost was a concern, but not an overriding one. Politics took center stage.

There were three distinct camps: new development, package, and modernization. All three camps had hardened and seasoned politicalconstituents. Each camp presented valid reasons why the organization should go with its preferred method. Management was anxious
to get the project started, but wanted everyone to feel good about the decision. We were brought in to break up the logjam and help aid
in the decision process. We used several of our specialized tools to come up with our analysis.
Our CHAOS Project database gave us the cost, risk, gain, and reasons for success and failure. We were able to match all three methods
against the profile, with a 90% match of the 120 attributes of more than 100 projects. Since 1994, The Standish Group has been studying
research project success factors and best practices. We have 10 factors of success and each success factor has 10 best practices. We’ve
assigned each best practice with weighted points. A perfect score would be 100 points, however the top 30 best practices add up to 50
points. Our TCO database gave us the operating costs for the application running on commodity hardware versus the IBM z10. A special
CICS project database allowed us to verify the project cost.
The purpose of this report is to provide the reader with a fair comparison of differences between rewriting, buying a package, andmodernization of a software application. The comparison includes information on their cost, risk, and reward. This report is broken into
five sections: case comparison, new development, application package, modernization, and proof points. Much of this report is basedon the CHAOS Research project on the creation and implementation of application software. The Standish Group has been the leading
provider of project management research and reporting since the introduction of the first CHAOS Report in 1994. These 16 years of
cumulative CHAOS research encompass more than 70,000 cases of completed IT projects. Through Standish Group’s CHAOS Research
programs, we have hosted more than 500 workshops, as well as countless one-on-one interviews, focus groups, project assessment
sessions, and executive retreats. The current CHAOS report is an online resource and is continually updated. As of the writing of thereport, the organization had not made a decision. We were not asked for and are not making any recommendations at this point, but the
information was presented to help the organization make the most informed decision possible.

AFTER YOU READ THIS REPORT, WHAT WOULD YOUR DECISION BE?

MAINFRAME VS. COMMO D I T YBASIC COST ($000)

Hardware Cost

MAINFRAME

COMMODITY

$321

$73

DIFFERENCEDIFFERENCE($248)-77%

Software Cost

$192

$26

($166)-86%

Manpower Cost

$645

$85

($560)-87%

Maintenance Cost

$132

$56

($76)

-58%

Other Cost

$282

$53

($229)-81%

Total Basic Cost

$1,572$293($1,279)-81%

Basic Cost

$1,572$293($1,279)-81%

Software Infrastructure

$400

$201

($199)-50%

Database & Systems
Admin

$554

$332

($222)-40%

Application Maintenance

$926

$314

($612)-66%

Other Cost

$218

$90

($128)-59%$3,670$1,230($2,440)-66%

APPLICATION COST ($000)

Total Operating Cost

Copyright © 2010 The Standish Group International, Inc.

OPERATIONAL COST OFOWNERSHIP
The table shows the current cost of themainframe and the estimated cost of thecommodity platform running the orderprocessing application. The operational costfor each of the methods is relatively the same.
We will be using this operational cost for all
three cases for the TCO and ROI analysis.
This represents a one-year operating costusing a three-year lease/write-off scheme
for software and hardware. Please note: The
cost to operate the commodity platform totest and run in parallel is added to the ROI
calculation for each case.
Note: All currency amounts in this report are in
thousands of US dollars ($000).

3C A S E C O M PA R I S O N

I

n comparing the resolution results of all three cases,
modernization stands out as having the highest chanceof success and the lowest chance of failure. Application
package implementation is also favorable to success, but more
likely it will be a challenged project. However, the package
implementation is twice as likely to fail as the modernization
project. The largest risk for the organization is the new software
application development. It is six times more likely to fail thanthe modernization project and three times more likely to failthan a package implementation. New software development
projects of this size have a poor track record in general, andonly organizations with a very high-quality project environment
have success.

COST OVERRUN COMPARISONRESOLUTION

CASE 1:APPLICATIONDEVELOPMENT

CASE 2:PACKAGEAPPLICATIONCASE 3:
MODERNIZATION

Below 20%

43%

22%

46%

20% to 50%

21%

36%

29%

51% to 100%

10%

29%

14%

Over 100%

26%

13%

11%

Our Estimate

44%

47%

34%

Current Estimate$10,000$5,000$3,500Parallel Testing$1,230$1,230$1,230

Average Overrun

S4,000$2,000$1,000Estimatewith Overrun$15,230$8,230$5,730The cost overruns in all three cases.

RESOLUTION RESULTS COMPARISONCASE 1:APPLICATIONDEVELOPMENT

CASE 2:PACKAGEAPPLICATION

Successful

4%

30%

53%

Challenged

47%

54%

39%

Failed

49%

16%

8%

RESOLUTION

CASE 3:
MODERNIZATIONThe resolutions of projects in all three cases.
Breaking down the cost overrun, we see 75% of the challenged
modernization projects are over budget by 50% or less. In
comparison to modernization, 62% of the new development
and 58% of package applications had a cost overrun of
50% or less. In contrast, 42% of package applications haveoverruns greater than 50%. Only a quarter of modernization
projects have overrun rates greater than 50%, and 38% of new
development projects exceed overruns of 50%. Our estimatesindicate that the average cost overrun is: Case 1, 44%; Case 2,
47%; and Case 3, 34%.
In breaking down the time overrun, we see 80% of thechallenged modernization projects are 50% or under thetarget delivery date. In comparison to modernization, 47% of
the new development and 59% of package applications had
a time overrun of 50% or less. In contrast, 53% of application
development had time overruns greater the 50%. Only 20% ofmodernization projects had time overruns greater than 50%
and 41% of package application exceeded overruns of 50%. Ourestimates indicate that the average time overrun is: Case 1,
44%; Case 2, 45%; and Case 3, 29%.

TIME OVERRUN COMPARISONCASE 1:APPLICATION DEVELOPMENT

CASE 2:PACKAGE APPLICATIONCASE 3:
MODERNIZATION

28%

27%

59%

20% to 50%

19%

32%

21%

51% to 100%

30%

31%

12%

RESOLUTION
Below 20%

Over l00%

23%

10%

8%

Our Estimate

44%

45%

29%

Estimate in Months

36

24

18

Average Overrun

16

11

6

Estimate with Overrun

52

35

24

The time overruns in all three cases.

4

M O D E R N I Z AT I O N C L E A R I N G A PAT H WAY T O S U C C E S S

C A S E C O M PA R I S O N

I

n comparing the ROI history, we see that 52% of modernization projects reported high ROI, while only 11% of new application
development projects and 34% of package application projects reported high ROI. Almost a quarter of application development
projects reported low returns, versus 11% for modernization and 9% for package applications. Two-thirds of new development
projects had an average return versus 57% for packages and 37% for modernization.
Comparing the estimated return on investment among all
three cases, we see optimistically that Case 3: Modernization
has the lowest cost with the highest return. If we look at it
pessimistically, Case 1: Modernization still has the lowest
cost with the highest return. Case 3: New Development has
the highest cost and lowest return, in being both optimistic
and pessimistic. The payback period is extended well beyond
most organizations’ attention span and business strategies.
It is because of the long time frames that many of thesetypes of projects are canceled before they have a positive
resolution. Case 2: Package Application sits in the middle
between Case 1 and Case 3.

EXPECTED ROI COMPARISONCASE 1:APPLICATIONDEVELOPMENT

CASE 2:PACKAGEAPPLICATIONCASE 3:
MODERNIZATION

High

11%

34%

52%

Average

66%

57%

37%

Low

23%

9%

11%

ROI

The expected return in all three cases.

In general, the comparisons between modernization projects and new development or package application projects are bothstark and dramatic. Modernization projects are mechanical and suited for contemporary IT organizations and the state of theirproject management capabilities. Modernization projects avoid the harder project management activities such as gathering user
requirements and receiving the executive attention that new development projects necessitate. Modernization projects also avoid
the gap analysis, user retraining, process changes, code modifications, and user rejections that package applications endure. Withapplication modernization projects, user training is very minimal and generally there are no modifications. Further, users are
generally more receptive because their methods and processes are not disrupted. The objectives of modernization projects areclearly stated and easily measured.
In measuring the order processing modernization project against the CHAOS Factors of Success, The Standish Group sees manyreasons to be confident. The top three reasons projects fail or overrun (lack of user involvement, executive support, and clear
business objectives) will have little impact on this project’s outcome. Since the modernization project is contained to the IT
department, there is very little worry regarding emotional maturity issues. Scope is also contained to migration and testing. Likean agile process, the application grew with small iterative changes and user instruction during the last 25 years. The rest of the
CHAOS factors and best practices coincide with the strengths of the organization. The new development or package application
projects do not enjoy these qualities.

ESTIMATED ROI COMPARISONCOMPARISONOPTIMISTIC

Return on Investment

Costs

Case 1: New Development
Case 2: Package Application
Case 3: Modernization

PESSIMISTIC

5-Year Gain

Payback in Months

Costs

5-Year Gain

Payback in Months$11,230$0

68$6,120$6,49042$15,230$090$8,230$1,75555$4,730$9,82032$5,730$7,00040

The estimated ROI in all three cases.

Copyright © 2010 The Standish Group International, Inc.

5

C A S E 1 : NEW APPLICATIOND E V E L OPMENTThe team proposing development from scratch using modern tools and techniques completed the project profile and environmental
assessment for the proposed new order processing application. The Standish Group reviewed the project case and responses tothe clarification follow-up telephone call. We matched the case against our database of more than 70,000 projects. The results are astatistical picture of projects that closely matched the proposed new project. In addition, the project assessment was used to more fully
understand the project environment and to make observations from both the project database history and the proposed new project.
We were able to match the order processing case with 90% of the success attributes in more than 100 cases in the CHAOS database.
Project resolution results in Case 1 indicate that there is a 4% chance that the project will come in on time, on budget, and not lacking
critical features. Forty-seven percent of the projects that match the case were considered challenged, meaning they were completed,but not on time, on budget, or with all the expected quality and features. In less than half the projects (49%), the users either rejected
the application or the organization stopped the project before completion.
Twenty-one percent of Case 1’s challenged projects were over budget
by 20% to 50%. The Standish Group estimates that challenged projects
in Case 1 will have an average cost overrun of 35%, which would
increase the project cost by $3.5 million dollars. Further estimating
analysis indicates the cost of the application will be greater than the
average bid by about 20% to 40%, for a cost of $12 to $14 million. Suchan overrun will cause the risk to increase dramatically and the payback
period to grow from almost six years to seven and a half years. The time
will also expand, and the likelihood of losing the executive sponsor andkey staff and project support personnel increases.
Thirty-eight percent of challenged projects were late by less than
20%; another 19% were 20% to 50% late. Thirty percent of challenged
projects were 51% to 100% late. Thirteen percent of challenged projects
in Case 1 were over 100% late. The Standish Group estimates thatchallenged projects in Case 1 will have an average time overrun of 46%,
which would increase the project time from three to four years. This
would put much pressure on the return on investment. Challenged
projects within the Case 1 set had a feature deficiency rate of 26%.
Most organizations are looking for a fast payback period and highrates of return to justify new software development projects. One-thirdof organizations want payback within one year; 41% 4%want less than a
two-year payback. Nineteen percent will accept a three-year payback.
However, only 6% will tolerate a greater than three-year payback. In
Case 1, a two-thirds majority reported low actual return on investment
from their projects. Twenty-three percent reported an average ROI,
while only 11% reported high returns. For Case 1, the return on
investment does not begin until the project is complete, tested, and inoperation. That is estimated to be three years.

RESOLUTION RESULTS
SUCCESSFUL

4%
FAILED49%
47%
CHALLENGED
The resolutions of projects in Case 1.

COST OVERRUN> 100%

16%26%

49%

47%

10

51–100%

< 20%

30%
43%

29%

10%
54% 21%

Our analysis indicates that our client is optimistic that the project
would come in on budget and on time. If that were the case, the20–50%
payback period would be six to seven years from the start of the
The breakdown of cost overruns for challenged projects in Case 1.
project. Of course, this is a highly unlikely event. If you include parallel
processing and most likely cost overruns of $2 to $4 million, thepayback period could easily be a decade or more. There is a way the organization could mitigate these costs by drastically reducing
scope and adopting the agile style, but how this may affect the outcome is beyond this analysis. In any case, such reduced scope
would be less than the current order processing application and therefore have reduced user benefit.

11%

6

13%

14%

46%
16%
M O D E R N I Z A T I O N C L E A R I N G A P A T H W22%
AY T O S U C C E S S
30%
29%
29%

%

C A S E 1 : NEW APPLICATIOND E V E L OPMENTO

ver the past 25 years, the cost associated with softwaredevelopment has shifted from 80% of the cost for developersand 20% for nondevelopers, to 20% for developers and 80% for
nondevelopers. The growth in application complexity is the main
reason for this shift. Automatic tools have increased programmer
productivity, and offshore development has reduced coding cost by anorder of magnitude. However, overall costs have not come down, butrather have increased. Project management overhead, sophisticatedrequirements gathering, converting requirements into detailed
specifications, load and performance testing, and other activities have
driven up both scope and complexity. This, in turn, has driven up
overall project costs.

CASE 1: RETURN ON INVESTMENT
ITEM

TYPE

CURRENTPROPOSEDPERIOD

Application
Development

Cost

$0

$3,000

One Time

Parallel Testing

Cost

$0

$1,230

One Time

Computer
Operation

Cost$3,670$1,230YearlyOrder processing &Inventory control

Savings$0$1,800YearlyFive-Year Gain$0

Payback Period
Months68

There were so few successful cases that match the client’s project
profile that we could not come up with common success factors and
The estimated ROI with a 68-month payback and a five-year
correspond them with the assessment. Projects that were challenged
gain of zero. This assumes no cost overrun, but includes parallel
showed similar assessment scores to the organization, in the low
processing.
60s. Common factors for both the Case 1 project set and our client’sorganization included good scores in general project management
mechanics. Both also had good scores on financial and risk
management. However, the common stress factors among the failed projects in the Case 1 set and the client organization are troubling.
Both our client organization and the failed project set showed poor scores on the top three project success factors: user involvement,
executive support, and clear business objectives.
Based on CHAOS definitions, the client’s project has a high likelihood of failure or of being challenged with a large overrun.
Communication and collaboration scored weak and are a real liability. Stakeholder and executive sponsor management is another area ofvulnerability. Project, risk, and financial management are areas of strengths. While the budget for Case 1 is $10 million, nonbudget items
such as user time and lost productivity, executive time, and IT compliance could easily add another 35% to 40% of hidden costs. A projectof this size will require 5,000 to 10,000 decisions. The organization’s weak executive decision skill is the most glaring area of concern.

D E S I R E D P A Y B A C K P E R I ODDECISION DISTRIBUTION SKILLSPOORLYSKILLED> 3 YEARS

6%< 1 YEAR

6%
<3 YEARS

43%

19%

34%
19%

34%

12%9%
30%

41%

41%

1%

<2 YEARS

The expected payback periodthat executives demand for new software projects.

9%

34%

49%

34%

12%
30%

SKILLED

49%

MODERATELYSKILLED
The general skill level of executive sponsors in the
distribution of decision power to team members.

8%

9%

Copyright © 2010 The Standish Group International, Inc.

22%

9%

HIGHLYSKILLED

8%

7

C A S E 2 : COMMERCIALA P P L I C ATION PACKAGEA

s in Case 1, the team proposing to replace the current order processing system with a commercial off-the-shelf (COTS)
package with moderate modifications completed the project profile and environmental assessment. The Standish
Group reviewed the project case and responses to our clarification follow-up telephone call. We matched the case against
our database of more than 70,000 projects. The results are a statistical picture of projects that closely matched the proposed
new project. In addition, the project assessment was used to more fully understand the project environment and to makeobservations from both the project database history and the proposed new project.
We were able to match the COTS order processing case with 90% of the success attributes in more than 100 cases in the
CHAOS database. Project resolution results in Case 2 indicate that there is a 30% chance that the project will come in on
time, on budget, and with most of the critical features. Fifty-four percent of the projects that match the case were consideredchallenged, meaning they were completed, but not on time, on budget, and with the expected quality and features. There is a
16% chance the users will either reject the application or the organization will stop the project prior to completion.
Thirty-six percent of Case 2’s challenged projects were over budget by 20% to 50%, while 29% were over budget by 51% to100%. The Standish Group estimates that challenged projects in Case 2 will have an average cost overrun of 39%, which would
increase the project cost by $2 million. Further estimating analysis indicates the cost of the COTS application will be around$8.2 million. Such an overrun will cause the risk to increase and the payback period to grow from three and a half years to overfour and a half years. The time will
also expand, and the likelihood of losing the executive sponsor and key staff and project4%
4%
support personnel increases.

26% 26%

Twenty-seven percent of challenged projects were less than 20% late; another 32% were 20% to 50% late. Thirty-one percent
of challenged projects were 51% to 100% late. Ten percent of challenged projects in Case 2 were over 100% late. The Standish
Group estimates that challenged projects in Case 2 will have an average time overrun of 47%, which would increase theproject time from two years to two and a half years. Again, this would put much more pressure on the return on investment.
Challenged projects in the Case 2 set have no feature deficiency, but the average organization will make 30% to 40% businessprocess changes.

49% 49%10% 10%47% 47%21%

R E S OL U T I O N R E S U L T S
FAILED

16%

>100%

13%

30% 30%13%
22% 22%

BELOW
20%

9%

9

57% 57%

36% 36%

54% 54%

51–100%20–50%The breakdown of cost overruns for challenged projects in Case 2

The resolutions of projects in Case 2.

11%
8

19

21%

29% 29%CHALLENGED

19%

COST OVERRUN

SUCCESSFUL

16%

43% 43%

6%

11%

11%

11%

14% 14%
M O D E R N I Z A T I O N46%
C L E A46%
R I N G A PAT H WAY T O S U C C E S S
52% 52%
37% 37%29% 29%

43%

C A S E 2 : COMMERCIALA P P L I C ATION PACKAGE

I

n looking at the Case 2 examples, we see 34% reporting high ROI. Fifty-seven percent reported an average ROI, while only 9%
reported low returns. For Case 2, the return on investment does not begin until the project is complete, and that is estimated to
be two years. Our analysis indicates that if the organization is optimistic and the project comes in on budget and on time, then the
payback period would be 68 months from the start of the project. Of course, this is not a sure thing. If you include parallel processing
and mostly likely cost overruns of $1 to $2 million, the payback period could easily be another year or two.
There are some major activities in a package implementation that cause many significant decisions; these are around what featuresare selected, parameters, data conversions from existing database sources, code modifications, business process changes, anduser training. In any package implementation there is always user conflict around changing the application code or the businessprocess. One of the reasons we see great success in package implementations for start-ups or new divisions is because there is not
an entrenched process. With mergers, we have seen major delays and problems around an organization’s attempt to have a single
operational process for common items such as ordering, inventory, customer service, and other business areas.
Common success factors among the Case 2 projects and our client include project management expertise, risk, and financial
management. One major common success factor is the alignment between the application software package and the current
business process. While moderate modifications are planned, keeping them to absolute essentials will increase the overall chancesof success and minimize overruns. This seems to be an area of strength for this project and is a very good indicator of success.
However, the biggest reason for project success is a speedy decision process. Successful projects were led by an executive sponsor
and stakeholders who made rapid decisions. In contrast, this is a major problem for our client.

6%an average assessment score in the 70s, and the assessment score for our client’s COTS teamProjects that were successful had
was 55, which is 5 points lower than the application development team. Common stress factors for challenged and failed projects
in Case 2 again showed poor scores on the top three success factors: user involvement, executive support, and clear businessobjectives. The organization scored low in the soft skills: stakeholder management, change management, and requirements
understanding.

19%

9%

34%

12%

30%

Based on CHAOS definitions, the Case 2 project has a high likelihood of being challenged with a moderate overrun. Success seems
well within the realm of possibility, and outright failure is remote. As in Case 1, stakeholder and executive sponsor management
is an area of vulnerability. Project, risk, and financial management are areas of strength. While the budget for Case 2 is $5 million,
nonbudget items such as user time and lost productivity, executive time, and IT compliance could easily add another 50% to 60% of
hidden costs. A project of this size will require 3,000 to 5,000 decisions. Again, the organization’s weak executive sponsor decisionskills is the most glaring area of concern and will most likely cause the project to overrun both the budget and time.

41%

49%

CASE 2: RETURN ON
INVESTMENTROI EXPERIENCELOW ROI
ITEM

9%

%

34%
AVERAGE
ROI

HIGH
ROI

57%
The expected experience payback
within the Case 2 set of projects.

CURRENTPROPOSEDPERIOD

Cost

$0

$500

One Time

Implementation

Cost

$0

$4,500

One Time

Parallel Testing

Cost

$0

$1,230

One Time

Computer Operation

39%

Cost

YearlySavings

53%$0$1,230Order Processing &Inventory Control$1,800YearlyFive-Year Gain

$6,490

Payback Period
Months42$3,670The estimated ROI with a 42-month payback and a five-year gain of$6.5 million. This assumes no cost overrun, but parallel processing.

11%

20%

Copyright © 2010 The Standish Group International, Inc.

52%

8% TYPE

Package50%

9C A S E 3: M O D E R N I Z AT I O NAs with Case 1 and 2, the team proposing to modernize the current order processing application with added features andmoderate modifications completed the project profile and environmental assessment.The Standish Group reviewed the
project case and responses to our clarification follow-up telephone call. We again matched the case against our database of morethan 70,000 projects. The results again gave us a statistical picture of projects
4% that closely matched the proposed new project.
In addition, the project assessment was used to more fully understand the project environment and to make observations fromboth the project database history and the proposed new project.

26%

49%

43%

We were able to match the order processing modernization case with 90% of the success attributes in more than 100 cases in
the CHAOS database. Project resolution results in Case 3 indicate that there is a 53% chance that the project will come in on
time, on budget, and with most of the critical features. Thirty-nine percent of the projects that match the case were consideredchallenged, meaning they were completed, but not on time, on budget, or with the expected quality and features. There is an 8%
chance the users will either reject the application or the organization will stop the project prior to completion.

10%

47%

21%

Seventy-five percent of challenged projects that matched in Case 3 were under budget by 50% or less. Forty-six percent
were under 20% overrun. The Standish Group estimates that challenged projects in Case 3 will have an average cost overrunof 36%, which would increase the project cost about a million dollars. Further estimating analysis indicates the cost of themodernization project will range from $3.5 to $4.5 million. Such an overrun will cause some risk to increase and the payback
period to grow from less than three years to three and a half years. The time will also expand, and the likelihood of losing theexecutive sponsor and key staff and project support personnel is increased slightly.

9%

%

12%

Fifty-nine percent of challenged projects were less than 20% late; another 21% were 20% to 50% late. Twelve percent ofchallenged projects were 51% to 100% late. Eight percent of challenged projects in Case 3 were over 100% late. The Standish
Group estimates that challenged projects in Case 3 will have an average time overrun of 30%, which would increase the projecttime from 18 to 24 months. This would have an effect on the return on investment. Challenged projects in the Case 3 set have no
feature deficiency.

30%

13%

16%

49%

29%
36%

54%
RESOLUTION RESULTS

COST OVERRUN>100%

FAILED

8%

39%

22%

30%

SUCCESSFUL

51–100%

11%
14%

53%
20–50%

CHALLENGED
The resolutions of projects in Case 3 .

BELOW
20%

46%

29%

The breakdown of cost overruns for challenged projects in Case 3.

20%
M O D E R N I Z AT I O N C L E A R I N G A PAT H WAY T O S U C C E S S

10

50%

30%

13%

16%

30%

9%

22%

C A S E 3: M29%
O D E R N I Z AT I O N
36%

54%

I

n looking at the Case 3 projects, we see 52% reporting high ROI.
Thirty-seven percent reported an average ROI, while only 11%
reported low returns. For Case 3, the return on investment does not
begin until the project is complete, and that is estimated to be one
and a half years. Our analysis indicates that if the client organization
is optimistic and the project comes in on budget and on time, then the
payback period would be 31 months from the start of the project. Of
course, this is not a sure thing. If you include parallel processing andmost likely cost and time overruns, the payback period could easily be
another four to five months.

46%

29%

57%

ROI EXPERIENCELOW ROI

11%

14%

34%

11%
AVERAGE
ROI

37%

HIGH
ROI

52%

There are generally two phases to modernization projects: migrationand enhancement. The major activities in a modernization migration
phase are mostly mechanical. Major activities include code conversion,
data source conversion, refactoring, rehosting, quality control, andExpected experience payback within Case 3 projects.
testing. Most notable are the activities that are not performed, suchas requirements documentation and user involvement, executive
sponsorship, steering committee meetings, stakeholder meetings, user
training, and hundreds of other time-consuming details. There are few decisions required, and those are focused on the mechanics andtechnical areas. The estimate for the migration phase is nine months. The enhancement phase is more like that of microprojects. The
first two microprojects are the real-time inventory update and compliance modules. Other enhancement activities include: improving
the user experience, replacing small modules, integrating open source, improving security, and other components.
Common success factors among the Case 3 projects and the client include project management expertise, risk, and financial
management. The client scored highly for technical skills, which is a key attribute for these types of projects. Projects that weresuccessful had an average assessment score in the 50s, and the assessment score for the client team was 58, points lower than the
package team (60 points) and higher than the application development team (55 points). The reason that organizations with lowscores can succeed in a modernization project is they avoid the key failure items. The biggest factors of success are user involvement,
executive support, and clear business requirements, which account for 50% of the assessment scores.
There were no common stress factors for challenged and failed projects among Case 3 and the organization. Challenged andfailed projects showed poor mechanics execution, including project management expertise, skilled resources, and poor tools andinfrastructure. These are strengths of the organization, not weaknesses. The other reasons for challenged and failed projects areoverreaching, data quality, and unusual languages. These do notseem to be an issue for our client. Overreaching of added features
CASE 3: RETURN ON INVESTMENT
and requirements is the most dangerous obsession. This obsession
ITEM
TYPE
CURRENT PROPOSED PERIOD
can turn an easy and straightforward project into a difficult andMigration
Cost$0$3,000
One Time
complex one. The organization has to watch out not to overreach.
Upgrade

Cost

$0

$500

One Time

Parallel Testing

Cost

$0

$1,230

One Time

Computer
Operation

Cost$3,670$1,230YearlyOrder Processing &Inventory Control

Savings$0$1,800YearlyFive Year Gain

$9,820

Payback
Period Months

32

The estimated ROI with a 32-month payback and a five-year
gain of $9.8 million. This assumes no cost overrun, but parallel
processing.

Copyright © 2010 The Standish Group International, Inc.

Based on CHAOS definitions, the Case 3 project has a highlikelihood of being successful or challenged with a small overrun.
Failure is only a remote possibility. The organization’s project
management and technical skills are outstanding. The orderprocessing modernization project characteristics favor the moremechanical project environment skills. The budget for Case 3 is$3.5 million. There are very little nonbudget items, such as user
time and lost productivity, executive time, and IT compliance. We
estimate that nonbudget items for Case 3 add about 10% to 15% of
hidden costs. A project of this size and type will require only about
200 to 300 decisions. This limited number of decisions favors a
positive outcome.

11

PROOF POINTSPROJECT
ASYST
Applied Industrial Technologies is a leading industrial distributor of bearings, power transmission components,and other industrial products. OMNEX is a homegrown enterprise distribution application. Fifteen years ago, thisapplication was developed by a small staff and quickly became the core of AIT’s IT services. During the next 15
years, AIT incrementally added programs and functionality to rival the likes of SAP’s or Oracle’s ERP distribution
systems. However, the system was tailored to the exact business operations and business processes of AIT. The
application was written in COBOL; it was developed for use by industrial sales, specialized using green screen
technology. The sales professionals adapted quickly to the original system because it had limited functionality andslowly added new functionality. However, new sales associates found the system difficult and intimidating.
In 2005, AIT set out to modernize OMNEX and started a project call Asyst. The focus of the project was to replace
the green screens with a modern graphical user interface (GUI) and provide integration with the company’s web
presence for the sales professionals. Five developers worked on the project for less than one year, and the firstmajor rollout occurred in late 2005. Applied continued adding incremental functionality, and the project wascompleted in the spring of 2010, slightly ahead of schedule and under projected costs. The new user experience cut
the learning time for new sales associates from two years to two weeks.

CHRYSALISPROJECT
The Bureau of National Affairs (BNA) is an independent publisher of legal analysis. Its subscription and billing
management systems did not reflect the current business. The system was designed around a print delivery method,which BNA grew over 50 years. Changes, such as web delivery, were difficult to make and took a long time. It
required more than 100 customer service representatives to support the current application. BNA had 250 marketingpeople selling its products. The current system did not let the salespeople enter orders, check status, or generateleads. Rather than modernize the existing application, BNA’s solution was to build a brand-new system from scratchusing modern development tools and waterfall project management techniques.
BNA tried four times to replace the subscription and billing management system. Each time the project ended incancelation and failure. There were few things done right in the project from a project environment standpoint.
Granted, BNA had a staff that knew the technology and could develop and support it. And the organization had
good project managers and many of the right project tools. But that was not enough to make it successful. The
organization had a very poor environment to execute projects. BNA also had a tendency to continually increase
scope, delay decisions, and stray from its core principles.

12

M O D E R N I Z AT I O N C L E A R I N G A PAT H WAY T O S U C C E S S

PROOF POINTSPROJECTSERVICEiRobot, a leader in robots for military and law enforcement, had no automated way of tracking what types offailures were plaguing the robotic units that were being sold to end users. Additionally, it was difficult if not near
impossible to know who had what unit and where it was. These robotic units were being sold into the governmentand deployed with various military combat units. Additionally, soldiers would field strip some units and swap parts
to make repairs while they were on missions. Then they would send the robot along with boxes of parts back to thecompany’s service organization to determine what caused the failure.
The solution was to implement Oracle Service Call and Depot Repair. The project plan was established, which
included milestones for specific functions, as well as lunch and learns for the end users to be taught as the projectprogressed. The VP in the government division was the executive sponsor. The consultants’ jobs were to do thesoftware installation and make any small changes that may be necessary, and conduct the lunch and learns eachweek through the duration of the project. However, by Standish measurements the project failed. The project failed
because of three of the Five Deadly Sins: Abstinence, Arrogance, and Ignorance.

PROJECT
MODERNIZATION
In 1999, the State of North Dakota formally created the Information Technology
Department (ITD) from the Central Data Processing Division and other departments within the state’s executive
agencies. The purpose is to support the state’s IT and communication requirements and to enjoy the benefits of aconsolidation of efforts. In 2004, ITD decided to get rid of its IBM mainframe and move to commodity servers mainly
because of the high cost of software from IBM, CA, and other mainframe software suppliers. In order to replace themainframe with commodity servers the state would need to migrate more than 100 mainframe applications.
The State of North Dakota had a variety of online and batch mainframe applications throughout its many agencies
and departments. Most of the mainframe applications were developed using Natural and COBOL languages. The
project modernization was scheduled to take two years at a cost of $4 million. The project started in March of 2007
and was completed in September of 2008 at a cost of $3.6 million. One of the big surprises of the project was theease of converting the code to the commodity servers, which took much less time than planned. The other surprise
was that testing took much longer than planned.

Copyright © 2010 The Standish Group International, Inc.

13

PROOF POINTSGAMEPROJECTIn 1997, the Bellagio in Las Vegas was only a few months from opening. Steven Wynn wanted to replace the aging
hotel reservation system. He also wanted the hotel reservation system to be fully integrated with a new advanced
gaming application. A few years prior to the opening of the Bellagio, Mirage Resorts, Inc., started on the new
integrated hotel and casino project. The new application would be state-of-the-art using the latest in commodity
hardware and software. Mirage used advanced waterfall project management techniques and tools. It had a very
thick requirements document with modeling and prototyping. Mirage hired an all-star staff to develop the integrated
hotel and gaming application.
The project suffered from constant changes and increased scope elegance (creep). The underlying software
infrastructure and development tools, while state-of-the-art, were immature. The infrastructure and development
software were fraught with bugs, documentation errors, and diminished functionality. The project languishedand was restarted a couple of times. Staff was replaced, and executive sponsors either bailed out or became very
discouraged. However, no one spoke for the user, and in the end the product was unusable. At the end, Bellagio
opened with the same software that was running on Mirage Resorts’ other properties: the Mirage Hotel andTreasure Island.

ACUTE
PROJECT
Owens & Minor is one of the leading suppliers of acute-care medical and surgical products for the healthcare
industry. In the early 1980s, Owens & Minor purchased a distribution software package to run on a standard IBM
mainframe infrastructure including CICS and DB2. During the next 25 years the organization added new and needed
functionality until the system was a full-blown ERP system. However, by the mid-2000s the system needed a major
overhaul. For example, a customer representative may need to look through a dozen green screens just to find asimple answer for a client. This would task the system during peak times that required more processing MSUs at
an increased cost. In addition, the base rising costs of the IBM mainframe platform were making it too expensive tocontinue.
Owens & Minor looked at its options: The company could either upgrade the package to a new version or buy adifferent package. In either case, it would need to modify either the application or its business processes. The
company would also lose the 25 years of incremental changes, which was a real asset to the business. Owens &Minor decided to modernize the application and run it on commodity hardware and software. The project took three
years and cost $9 million. Owens & Minor reduced its operating costs by $6 million with an 18-month payback. More
importantly, customer service could provide answers in one screen and other users readily accepted the new system
as they were not impacted and required no training.

14

M O D E R N I Z AT I O N C L E A R I N G A PAT H WAY T O S U C C E S S

22%
29%
36%

34%

39%

I N S U M57%
MARY

FEATURES USEDIn 1996, we did a study on 100 custom development
applications to look at the functions and features requested,
implemented, and used. The study
11%was done in two steps.
STEP 1: We took an inventory of each feature and function.

46%

STEP 2: We held user workshops to find out which features
were used.

52%

53%

OFTEN
HARDLYEVER

We learned that only 7% of37%
features were always used, another
13% were often used, 16% were sometimes used, 19% wererarely used, and 45% were never used. The study was very long,difficult, and expensive. Using some automated tools and spot
checks every couple of years, we found, to no surprise, thatthe numbers are relatively unchanged for most methodologies.
Our current thinking is that 20% of features are often used and
50% of features are hardly ever or never used. The gray area isabout 30%, where features and functions get used sometimes
or infrequently. Requirements gathering, selecting, andimplementing is the most difficult task in developing custom applications.

20%
50%

30%
INFREQUENTLYThe Standish Group estimate offeatures used in custom applications.

Over the years we have also evaluated the features used in package applications. Here we found less than 5% of the featuresand functions get used. In implementing a package application there is enormous pressure to get the value from it through thejudicious use of these features and functions. There are literally thousands of decisions that have to be made during the life ofa package implementation project. Our research shows that for every $1,000 in project cost, the organization will need to make
1.5 decisions. A million-dollar project will produce 1,500 decisions, while a $5 million project will have 7,500 decisions. During
a typical medium-size ERP system implementation the organization will have to make more than 10,000 decisions. This is truly
misery for people who don’t like to make decisions.   
In contrast, modernization projects are pre-optimized. There are no process changes and little to no training. There are few
decisions, and most of those decisions are on the project mechanics and not on the more fuzzy or difficult feature/function
choices. There is little discord around requirements. For better or worse, the users get what they had before the modernization
project, but improved and less costly.
Now, it is time to make your recommendation decision. Should the organization go with Case 1: New Application Development,
Case 2: Package Application Implementation, or Case 3: Modernization? Please let us know your decision by e-mailing
Jennifer@standishgroup.com.

MODERNIZATION: CLEARING A PATHWAY TO SUCCESS is based on a real customer engagement. The clientorganization completed three separate project assessments by three individual teams. The project assessment procedure asks
about 120 questions with a follow-up interview. The Standish Group then measured this assessment data against our CHAOS
database of more than 70,000 projects. These assessments are a standard service from The Standish Group. In addition to theindividual project assessment, The Standish Group provides project environment assessments, executive sponsor assessments, andproject management assessments. As of the writing of this paper, The Standish Group is looking at ways to speed up and improve
the project decision process and how IT coaches executive sponsors. All data and information in this report should be consideredStandish opinion, and the reader bears all risk in the use of this opinion. The Standish Group is available to assess your project andproject environment.

Copyright © 2010 The Standish Group International, Inc.

15

ABOUT THE VIRTUAL ADVISOR®
The VirtualADVISOR® System is designed to improve the efficiency and value of IT performance, while increasing the delivery speedof critical applications and IT infrastructure. The VirtualADVISOR® System is a collection of proven wisdom-based management
tools used by Standish Advisors (STARs) to help IT managers increase their understanding of their business and IT environment by
providing case-based and enterprise-wide alternative solutions. Using highly advanced case-based reasoning technology, STARs areable to profile your project, application, or systems for total cost of ownership (TCO), return on investment (ROI), and risk against
70,000 project cases and 2,000 system cases. You can apply these cases against more than 100 applications and uses with over 20
system types, seven database types, and many types of middleware, making the VirtualADVISOR truly virtual. There are several
major features that make the VirtualADVISOR an IT management system without comparison:

ASSETSPROJECTS+1,200 CIOs
2,000 Cases
8-12 Hours/Case
10 x 2K = 20K Hours
2,500 Days
12.5 CIO Years

5,000 IT/PMs
70,000 Cases
1-4 Hours/Case
2 x 70K = 140K Hours
18,000 Days
84 PM Years

Freshness: The case database is always fresh and up to date; nocase in the asset database is older than six months. Since the
VirtualADVISOR is a web-based thin-client tool, new data is addedand existing data is updated in the background. Therefore, eachtime a STAR runs a case for you, you are getting the most up-to-date
results.
No Assumption: It is our database of actual cases that drives theconclusions; there are no assumptions used. Your case profile will
match against cases in the database. Items in the database have beenthoroughly scrutinized and categorized for importance. Each item hasa dynamic weight to ensure the maximum relevance to your profile.
Currently we are experiencing a 90% match rate.

Extensive: Each TCO case in the database takes about 50 hours of work to complete, usually by several people. This equates to 56 IT
person-years of work for the initial database. Every six months we retire a number of cases and update all the rest, plus each month
we collect downtime information for about another 40 person-years of IT effort. The project case database, which we accumulate,
has about 50 IT person-years of work as well.
Diversity: While the VirtualADVISOR database represents more than 100 years of CIO, IT executive, and IT professional work, its
real strength is in its diversity of input. More than 20,000 people have contributed to the database across 100 countries, countless
industries, and a wide range of company sizes. This diversity of experience gives you a collection of expertise rather than one
person’s opinion or the opinion of a small group with their natural limited experience.
User Driven: The TCO was developed by user input; all data is collected from users and NO price data, performance, or any
other input is derived from vendors. We do not use benchmarks or consultant opinions to calculate cost or risk. This makes the
VirtualADVISOR a true unbiased source. The risk model was developed from 14 years of CHAOS University input and CHAOS dataof more than 70,000 projects.

ABOUT THE STANDISH GROUP INTERNATIONAL, INC.
Since 1985 The Standish Group, the leader in spotting future trends, has been helping end users
and vendors of technology solutions prepare for the future. The Standish Group delivers fast,consistent, reliable, independent IT advice built on a solid foundation of primary research. For
further information visit our website at: http://www.standishgroup.com.
The Standish Group International, Inc. • 60 State Street, Suite 700 • Boston, MA 02109, USA • P: +1.508.760.3600

16M O D E R N I Z AT I O N C L E A R I N G A PAT H WAY T O S U C C E S S

Copyright © 2010 The Standish Group International, Inc.