CHAOS REPORT 2015
T

he CHAOS Report 2015 is a model for future CHAOS Reports. There have only been two previous CHAOS Reports, theoriginal in 1994 and the 21st edition of 2014. This new type of CHAOS Report focuses on presenting the data in different
forms with many charts. Most of the charts come from the new CHAOS database from the fiscal years 2011 to 2015. The CHAOS
fiscal year starts March 1 and runs until the end of February. A few of the charts are from the new SURF database to highlightcertain information. The purpose of this report is to present the data in the purest form without much analysis and little thought
leadership. Analysis and thought leadership are offered in the CHAOS Manifesto series of reports.
Another major change is how we define success. We have multiple definitions, including our newest. We coded the new CHAOS
database with six individual attributes of success: OnTime, OnBudget, OnTarget, OnGoal, Value, and Satisfaction. Our Traditional
definition is OnTime, OnBudget, and OnTarget. This means the project was resolved within a reasonable estimated time, stayed
within budget, and contained a good number of the estimated features and functions. Our new Modern definition is OnTime,
OnBudget, with a satisfactory result. This means the project was resolved within a reasonable estimated time, stayed withinbudget, and delivered customer and user satisfaction regardless of the original scope. We have the flexibility to present theresults for one to six of these attributes in any combination.
ONBUDGETNO
56%

ONTIME

YES
44%

NO
60%

ONTARGET

YES
40%

NO
44%

The percentage of projects that were
OnTime from FY2011–2015 within the
new CHAOS database.

The percentage of projects that were
OnBudget from FY2011–2015 within the
new CHAOS database.

YES
56%

The percentage of projects that were
OnTarget from FY2011–2015 within
the new CHAOS database.

TRADITIONAL RESOLUTION FOR ALL PROJECTS

2011

2012

2013

2014

2015

SUCCESSFUL

39%

37%

41%

36%

36%

CHALLENGED

39%

46%

40%

47%

45%

FAILED

22%

17%

19%

17%

19%

The Traditional resolution of all software projects from FY2011–2015 within the new CHAOS database.
*All data, unless otherwise noted, represents results from FY2011-2015. The total numberof software projects is 25,000-plus, with an average of 5,000 per yearly period.
Copyright © 2015 The Standish Group International, Inc.

Our Modern Resolution definition is OnTime, OnBudget, with a satisfactory result. This definitionencompasses both a success rate for the project management of a project and for the project itself.
The Traditional Resolution of OnTime, OnBudget, and OnTarget clearly supports the goals of project
management, but not the customer or user of the product or project. The reason we consider thisto be the best definition is that it combines the project management process and the end resultsof a project. We have seen many projects that meet the triple constraints of OnTime, OnBudget,and OnTarget, but the customer was not satisfied with the outcome. This is evident in the data, whichshows a 7% decrease in the success rate and a 7% increase in the challenged rate.
In changing from the OnTarget constraint to satisfactory we avoid penalizing a project for having an evolving target, which all
projects have, even the very small ones. Customers have a clear opinion on the satisfaction level whether or not all the featuresand functions that they asked for in the beginning of the project are realized. In our research we found that both satisfaction
and value are greater when the features and functions delivered are much less than originally specified and only meet obvious
needs. In other research we found that most features and functions of software are not used. These additional features
increase cost, risk, and quality but do not necessarily provide value.
VALUABLENO41%

ONGOAL

YES
59%

NO
38%

SATISFACTORY

YES
62%

NO
44%

The percentage of projects that were
OnGoal from FY2011–2015 within the new
CHAOS database.

The percentage of projects consideredvaluable from FY2011–2015 within the new
CHAOS database.

YES
56%

The percentage of projects consideredsatisfactory from FY2011–2015 within the
new CHAOS database.

MODERN RESOLUTION FOR ALL PROJECTS

2011

2012

2013

2014

2015

SUCCESSFUL

29%

27%

31%

28%

29%

CHALLENGED

49%

56%

50%

55%

52%

FAILED

22%

17%

19%

17%

19%

The Modern Resolution (OnTime, OnBudget, with a satisfactory result) of all software projects from FY2011–2015 within the new CHAOS
database. Please note that for the rest of this report CHAOS Resolution will refer to the Modern Resolution definition not the Traditional
Resolution definition.

Copyright © 2015 The Standish Group International, Inc.

2

Project size has always been a major element in the CHAOS research. It was clear fromthe very beginning of the CHAOS research that size was the single most important factor in
the resolution of project outcome. On this page we show two tables: resolution of all software
projects by size; and size of the software projects by resolution. These tables clearly show the
impact of size on the results of Ontime, OnBudget, with a satisfactory result. It is also clear thatthe larger the project, the less valuable the return rate. In many cases larger projects never return
value to an organization. The faster the projects go into production the quicker the payback starts
to accumulate.
One of the major services of our Value Portfolio Optimization and Management Service is to break up large software
projects into multiple small projects, with early delivery for success, quicker return on value, and greater customer anduser satisfaction. We have found that mostsoftware projects only require a small team
PROJECT SIZE BY CHAOS RESOLUTION
for a short duration in order to deliver value
to the organization; only in very rare cases
SUCCESSFUL CHALLENGED
FAILED
TOTAL
do projects need to be larger and longer.
Most, if not all, large, complex, multi-year
projects are unnecessary. This is especially
6%
51%
43%
100%
Grand
true for standard infrastructure software—such as middleware, databases, and system
11%
59%
30%
100%
Large
management.
VALUE FOR LARGE PROJECTS
Medium

12%

62%

26%

100%

Moderate

24%

64%

12%

100%

Small

61%

32%

7%

100%

n Very High Value 4%n High Value 14%n Average 23%
n Low 16%n Very Low 43%

The size of software projects by the Modern Resolution definition from
FY2011–2015 within the new CHAOS database.

The return of value for large projects from FY2011–

CHAOS RESOLUTION BY PROJECT SIZE

to 2015 within the new CHAOS database.

SUCCESSFUL

CHALLENGED

FAILED

Grand

2%

7%

17%

Large

6%

17%

24%

Medium

9%

26%

31%

Moderate

21%

32%

17%

Small

62%

16%

11%

100%

100%

100%

VALUE FOR SMALL PROJECTSn Very High Value 17%n High Value 27%n Average 36%
n Low 9%n Very Low 11%

TOTAL

The return of value for small projects from FY2011–2015 within the new CHAOS database.

Copyright © 2015 The Standish Group International, Inc.

The resolution of all software projects by size from FY2011–2015 within
the new CHAOS database.
3

Looking at project resolution by industry provides another view of the CHAOS database. The
table on this page shows the resolution of all software projects by industry from FY2011–2015
within the new CHAOS database. The results show that retail projects had the highest success rate
at 35% using the Modern definition of success. The results also show that government projects had
the highest failure rate at 24%, and financial and government projects had the highest challenged
rate at 56%. The comparison of satisfaction level for banking versus retail shows that retail also has
better results.
Many of our Benchmark clients like to compare their results to other organizations in the same industry
and we do this as matter of course. However, we found that industry is not the most accurate or important metric of comparison.
The most accurate is to consider industry as a minor filter, with project type, size, skills, and methodology as primary filters.
Other minor filters would include organizational size and geography. In our Resolution Benchmark Membership we use thistechnique to benchmark project portfolios.
BANKING PROJECTSRETAIL PROJECTSn Very Satisfied 9%

n Very Satisfied 12%n Satisfied 21%

n Satisfied 27%

n Somewhat Satisfied 28%

n Somewhat Satisfied 32%

n Not Satisfied 19%

n Not Satisfied 18%

n Disappointed 23%

n Disappointed 11%

The satisfaction level for banking projects from FY2011–2015

The satisfaction level for retail projects from FY2011–2015
within the new CHAOS database.

within the new CHAOS database.

CHAOS RESOLUTION BY INDUSTRY
SUCCESSFUL

CHALLENGED

FAILED

Banking

30%

55%

15%

Financial

29%

56%

15%

Government

21%

55%

24%

Healthcare

29%

53%

18%

Manufacturing

28%

53%

19%

Retail

35%

49%

16%

Services

29%

52%

19%

Telecom

24%

53%

23%

Other

29%

48%

23%

The resolution of all software projects by industry from FY2011–2015 within the new CHAOS database.

Copyright © 2015 The Standish Group International, Inc.

4

In our CHAOS Report 2014 we showed that project resolution differed slightly by most of theareas of the world. In that report we used the traditional success metrics of OnTime, OnBudget,and OnTarget. In this report we used the Modern definition of success of Ontime, OnBudget, with asatisfactory result. We see a major bifurcation with North America and the rest of the world; North
America has a 31% success rate versus Europe at 25%, Asia at 22%, and the rest of world at 24%.
We have seen that North America has some of the highest emotional maturity skills. These skills
include managing expectations and gaining consensus, which in turn would cause a high satisfactionlevel. On the other hand, Asia has the lowest emotional maturity scores according to our emotional
maturity appraisals and benchmarks.
As we reported in the Factors of Success 2015 report, emotional maturity is the second-ranked Factor of Success. In that report
we stated that having a skilled emotional maturity environment helps 80% of projects enjoy success. The No. 1 Factor of Successis a skilled executive sponsor. These two factors, along with the other eight, are appraised as the first step in the Resolution
Benchmark. If we find during the skills appraisal that the executive sponsorships and/or emotional maturity skills are deficientthen we will provide advice on how to improve the score and help improve future Benchmark results. For more information on the
Factors of Success, please see the Factors of Success 2015 report.

BUDGET PROCESSPROJECT SELECTION PROCESSn Key executive sponsor sets then Only the boss/seniorbudget with project team 37%

executive 27%n Users set the budget withn There is a formal process

the project team 7%

to do it 50%

n All stakeholders are involvedn Each sector defines its
own priorities 21%

in the budget 31%

n The CFO sets the limit on the budget 17%

n It is very varied or I do not know 3%

n Other 8%

We asked the 37% of SURF respondents who said that the key
executive sponsor sets the budget with the project team, “Ingeneral, who participates in project selection/approval in yourorganization?” This is based on 111 responses in the SURF
database.

We asked IT executives, “What is your general practice on projectbudgeting and cost collaboration?” This is based on 300 responses
in the SURF database.

CHAOS RESOLUTION BY AREA OF THE WORLD

SUCCESSFUL

CHALLENGED

FAILED

North America

31%

51%

18%

Europe

25%

56%

19%

Asia

22%

58%

20%

Rest of World

24%

55%

21%

The resolution of all software projects from FY2011–2015 by the four major areas of the world.

Copyright © 2015 The Standish Group International, Inc.

5

The type of project has a major effect on resolution. The table on this page shows the resolutionof all software projects by project type from FY2011–2015 within the new CHAOS database using
the Modern definition of success. Projects using a purchased application with no modification
had the highest success rate at 57%. Projects that were developed from scratch using modern
methodologies had a 23% failure rate. This is the highest failure rate other than the “other”
category. The results also show that projects that were developed from scratch using traditional
languages and methods had the highest challenged rate at 61%.
Modernization projects had the second highest success rate at 53%. The Standish Group has a very
specific definition and development method for modernization projects. In fact, we modified “modernization” by adding “inplace” so as not to confuse the general modernization of applications by the other techniques such as developing from scratchusing modern methodologies or purchasing components. For more information on modernization in place, please see ourModernization in Place report.
GAIN VERSUS RISK METRICS

ROI FOR REQUIREMENTSn Calculate overall project andn Painful 31%

allocate over individual
requirements 15%

n Restrained 68%

n Calculate each requirementn Painless 1%

and add up to overall project
ROI 14%n Calculate overall project,calculate major requirements,
and allocate the rest 30%

n Do not calculate individual requirements 40%
We asked IT executives, “How do you calculate ROI for individual
requirements?” This is based on 300 responses in the SURF
database.

We asked the 40% of SURF
respondents who said they do not
calculate individual requirements,“How would you describe your organization’s efforts indeveloping and maintaining financial and risk metrics for projectrequirements?” This is based on 121 responses in the SURF
database.

CHAOS RESOLUTION BY PROJECT TYPE
PROJECT TYPE

SUCCESSFUL

CHALLENGED

FAILED

Developed from scratch using
traditional languages and methods

22%

61%

17%

Developed from scratch using modern
methodologies

23%

54%

23%

Developed some components andpurchased others

24%

59%

17%

Purchased components and assembled
the application

25%

59%

16%

Purchased application and modified

42%

37%

21%

Purchased application and performed
no modifications

57%

28%

15%

Modernization

53%

38%

9%

Other

28%

47%

25%

The resolution of all software projects by project type from FY2011–2015 within the new CHAOS database.

Copyright © 2015 The Standish Group International, Inc.

6

The table on this page compares the resolution of all software projects from FY2011–2015
within the new CHAOS database, segmented by the agile process and waterfall method. The total
number of software projects is more than 10,000. The results for all projects show that agile
projects have almost four times the success rate as waterfall projects, and waterfall projects
have three times the failure rate as agile projects. The results are also broken down by projectsize: large, medium, and small. The overall results clearly show that waterfall projects do notscale well, while agile projects scale much better. However, note that the smaller the project, thesmaller the difference is between the agile and the waterfall process.
As we stated in the Factors of Success 2015 report, we have identified two trump cards that together create a winning hand.
The trump cards are the agile process and small projects. As measured by Modern metrics, small projects using an agile
process only have a 4% failure rate. For more information on trump cards, please see the Factors of Success 2015 report.
TIME BOXESn Always 14%
n Yes, most of the time 23%
n Yes, some of the time 32%
n No 31%
We asked IT executives, “In general, do you utilize timeboxes to optimize your projects? This is based on 300
responses in the SURF database.

CHAOS RESOLUTION BY AGILE VERSUS WATERFALLSIZEAll Size
Projects

Large Size
Projects

Medium Size
Projects

Small Size
Projects

METHOD

SUCCESSFUL

CHALLENGED

FAILED

Agile

39%

52%

9%

Waterfall

11%

60%

29%

Agile

18%

59%

23%

Waterfall

3%

55%

42%

Agile

27%

62%

11%

Waterfall

7%

68%

25%

Agile

58%

38%

4%

Waterfall

44%

45%

11%

The resolution of all software projects from FY2011–2015 within the new CHAOS database, segmented by theagile process and waterfall method. The total number of software projects is over 10,000.

Copyright © 2015 The Standish Group International, Inc.

7

We use two tables to determine and appraise complexity. There are five attributes ineach table. We then add up the points based on the attributes of the project to determine thecomplexity level. A complexity level is entered for each project in the CHAOS database. We also
use the level in the Size-Complexity Matrix. Our Size-Complexity Matrix provides a guideline forcategorizing a project either by size or complexity.
For more information about the Size-Complexity Matrix, please review prior CHAOS Manifestos
including CHAOS Manifesto 2014. In addition, The Standish Group’s Portfolio Optimization andManagement Service can help develop strategies for reducing complexity and gaining success.
Complexity is one of the main reasons for project failure. The table on this page shows the resolution of all software projects
by complexity from FY2011–2015 within the new CHAOS database using the Modern definition of success. The results showthat 38% of very easy projects were successful. Very complex projects have both the highest challenged (57%) and failure(28%) rates. Inside of every complex problem are simple solutions. Complexity is often caused by size, conflicting goals, large
budgets, and executive sponsor egos.
LARGE, COMPLEX PROJECTSCOMPLEXITY APPRAISALn Yes, for all projects 6%

n Successful 2%

n Yes, for most projects 35%

n Challenged 42%

n Yes, for few projects 26%

n Failed 56%

n No 32%
We asked IT executives,“Does your organization develop
any appraisal of complexity on its projects?” This is based on
300 responses in the SURF database.

The resolution of large andcomplex software projects from
FY2011–2015 within the new CHAOS database.

CHAOS RESOLUTION BY COMPLEXITY

SUCCESSFUL

CHALLENGED

FAILED

Very Complex

15%

57%

28%

Complex

18%

56%

26%

Average

28%

54%

18%

Easy

35%

49%

16%

Very Easy

38%

47%

15%

The resolution of all software projects by complexity from FY2011–2015 within the new CHAOS database.

Copyright © 2015 The Standish Group International, Inc.

8

The Standish Group has stated for many years that clear goals are achieved when all the
stakeholders are focused on and understand the core values of the project. We believed that goalclarity and focus were essential to a successful project. However, measuring success by both the
Traditional and Modern metrics we found the opposite to be true. We coded the database with a
5-point scale, from precise to distant, in order to measure the effect on success rates. It is clear
from the research that goals closer to the organization’s strategy have the opposite effect onhigher satisfaction and success rates.
The Standish Group uses goal as one of the seven constraints as part of the Optimization Clinic. The
Optimization Clinic is the third step in our Value Portfolio Optimization and Management Service. We also use goal as one of
the measurements for our Resolution Benchmark. The Standish Group is now suggesting that your organization take action
over trying to achieve clarity. Many of the most satisfying projects start out as vague. The business objectives are dynamic as
the project progresses. Project teams should reduce or give up control of the business objectives to encourage and promote
innovation.

COST/BENEFIT ANALYSIS

LOW-VALUE BUSINESS PROCESSESn Yes, we have/had an Yes, we have/had aconcentrated effort 23%

concentrated effort 33%

n Yes, but it isn Yes, but it isopportunistic 31%

opportunistic 38%

n No, but in our plan 18%

n No, but in our plan 17%

n No plans 28%

n No plans 12%

We asked IT executives: “Do you have an active program
to optimize business processes by eliminating low-value businessprocesses?” This is based on 300 responses in the SURF
database.

We asked IT executives, “Do you have an active program to
optimize business processes by doing cost/benefit analysis for
new business processes?” This is based on 300 responses in the
SURF database.

CHAOS RESOLUTION BY GOAL
SUCCESSFUL

CHALLENGED

FAILED

Precise

22%

53%

25%

Close

23%

54%

23%

Loose

27%

52%

21%

Vague

38%

46%

16%

Distant

34%

58%

8%

The resolution of all software projects by goal from FY2011–2015 within the new CHAOS database.

Copyright © 2015 The Standish Group International, Inc.

9

Successful projects need smart, trained people. Not surprisingly, one of the key projectsuccess factors identified in Standish Group’s CHAOS research is a competent staff. There are
five key fundamentals to ensure staff competency. First, identify the required competencies andalternative skills. Second, provide a good, continuous training program to enhance the staff
skills. Third, recruit both internally and externally to provide a balance of experiences. Fourth,provide incentive to motivate the staff. Finally, ensure the staff is project-focused. When a project
has both teamwork and skilled resources, it can prevail under even the direst of circumstances. To
ensure a competent staff you must match the skills of the team to correspond with the needed skillsof the project. This is one of the seven constraints we use to prioritize your project portfolio.
Here we look at project resolution by capability. The table on this page shows the resolution of all software projects bycapability from FY2011–2015 within the new CHAOS database using the Modern definition of success. The results show that
38% of gifted resources were on successful projects. Projects that had unskilled people had both the highest challenged(60%) and failure (23%) rates. The decisions around project priority include: Do you go forward with a project if you lackskilled capability? This decision is especially pertinent for large projects with a large staff who have a mix of good and poor
resources. This is one of the reasons that small projects have a higher success rate since small projects are easier to staff
with high-performing teams.
GIFTED AGILE TEAMSUNSKILLED AGILE TEAMSn Very High Value 17%

n Very High Value 4%n High Value 27%n High Value 5%n Average 36%n Average 19%

n Low 9%n Low 44%n Very Low 11%

n Very Low 28%

The value rating of medium to large original software development
projects with gifted agile teams from FY2011–2015 within the new
CHAOS database.

The value rating of medium to large original softwaredevelopment projects with unskilled agile teams from FY2011–2015 within the new CHAOS database.

CHAOS RESOLUTION BY CAPABILITY
SUCCESSFUL

CHALLENGED

FAILED

Gifted

38%

45%

17%

Talented

31%

53%

16%

Competent

28%

53%

19%

Able

24%

54%

22%

Unskilled

17%

60%

23%

The resolution of all software projects by capability from FY2011–2015 within the new CHAOS database.

Copyright © 2015 The Standish Group International, Inc.

10

As we wrote in the Factors of Success 2015 report, the table below reflects our opinion of theimportance of each attribute and our recommendation of the amount of effort and investmentthat should be considered to improve project success. It is clear to us that the four primary
investments should be focused on: executive sponsorship skills, emotional maturity environment,user involvement, and optimization services. It is our tradition to assign points to each factor tohighlight its relevance. These points should also be considered as an investment guideline forproject management improvement. For example, if you are spending $50 million on IT projects then
2% of the money should be going toward improving the value of those projects. Based on this amount,
our recommended breakdown of money to be allocated to each factor is calculated next to the point value on
the chart. So, if you want your projects to be more successful, with higher value and greater customer satisfaction, you shouldcarefully consider where you invest your project improvement money.
The law of diminishing returns states that in all productive processes, adding one or more factors of production, while
holding others constant, will at some point yield lower returns. Project management expertise, process methods, and tools
are affected by the physics law of diminishing returns. The proof point of this law is the United States government. The U.S.
government has the highest level of project management expertise, the most sophisticated tools, and the highest level ofgovernance. Yet, U.S. government projects cost exponentially more than commercial projects and have greater failure rates.
While the Factors of Success can help improve project performance, the key to project management success is moderation.

CHAOS FACTORS OF SUCCESSFACTORS OF SUCCESSPOINTSINVESTMENT

Executive Sponsorship

15

15%

Emotional Maturity

15

15%

User Involvement

15

15%

Optimization

15

15%

Skilled Resources

10

10%

Standard Architecture

88%Agile Process

77%Modest Execution

6

6%

Project Management Expertise5

5%

Clear Business Objectives

4

4%

The 2015 Factors of Success. This chart reflects our opinion of the importance of each attribute and our recommendation of the amount ofeffort and investment that should be considered to improve project success.
Copyright © 2015 The Standish Group International, Inc.

11

ADDITIONAL RESOURCES AND RESEARCH
Other available research reports:• Factors of Success 2015
• Modernization in Place• CHAOS Manifesto 2014
Many of the subjects within this report are subjectsof CHAOS Tuesday, our Internet radio program. Theseshows include:• CHAOS Report 2014
• Exceeding ValueCHAOS Tuesday #81 CRAPOLACHAOS Tuesday #78 Success Redefined
CHAOS Tuesday #77 Factors of Success
CHAOS Tuesday #76 Haze

RESOLUTION BENCHMARKOur Resolution Benchmark Membership measures your project portfolio against like organizations with a similar project mix
and skill maturity.

MEMBERSHIP BENEFITS• Count what counts• Improve customer and user
satisfaction

Copyright © 2015 The Standish Group International, Inc.

• Increase project value• Reduced project overhead• Improve project environment

12

VALUE PORTFOLIO OPTIMIZATION AND MANAGEMENT SERVICE
Our Value Portfolio Optimization and Management Service is a forward-thinking and predictive visualization of thevalue of your software investments.

STEP 1: GETTING TO KNOW YOU
By focusing your project portfolio on value, our service frees your organization to create value.
Our service offers the following benefits:• High Returns on Investment• More Innovations• Greater Stakeholder Satisfaction• Less Management Frustration• Reduced Project Overhead

STEP 2: PROJECT PROFILESThe Value Portfolio Optimization and Management Service offers the following features:• Rapid• Simple• Comprehensive• Inexpensive• Comprehendible

STEP 3: OPTIMIZATION CLINIC
The one thing we are not going to do is immediately change your process or try to sell you new and cumbersome tools.
It really does not matter where you are in project management maturity. Our aim is to reduce or minimize the burden
placed on you by expensive tools and complex processes. We do this through our three unique items:• Research database of 50,000 projects• Patented optimization formula• Our insight into a broad set of projects

Our three-step method helps you focus on things that really count.
• STEP 1: Project Skills and Environmental Orientation• STEP 2: Individual Project Optimization & Assessments• STEP 3: Value Optimization Clinic
FOLLOW-UP: Each quarter The Standish Group will visit your organization to update your portfolio. Our Value
Portfolio Optimization and Management Service helps you exceed and create value.
Copyright © 2015 The Standish Group International, Inc.

13