The Real Reasons Why Technology Projects Fail and How to Fix It
By: Vasa Digital Architects
Tags: Enterprise Software, Enterprise Technology, Project Management, Software Development, Digital Transformation

About 70% of technology projects fail. Best practices, skills and competencies have been improving for decades — so why do project still fail? Projects fail when, they solve the wrong problem, can’t find a good enough solution or over invest because of the psychological biases.

Table of Contents

Summary

About 70% of IT / digital transformation projects fail. Approximately 45% are late and 25% get cancelled. The commonly held belief is that projects fail because they do not follow best practices and don’t have all the right skills on the team. However, the adoption of best practices, skills, and competencies have been improving for decades — so why do projects still fail?

The prevalent view is that that projects fail because they don’t follow best practices such as:

Better control over changing scope and requirements

Better project management practices

Use of agile and short delivery cycles

Managing strategy and stakeholders

Better control over technology complexity

Building better teams and improving communication

More rigorous quality assurance

Projects that fail often have unclear objectives, shifting requirements, struggle to use technology correctly, suffer from poor team dynamics, or often missing critical skill sets and lack strong project management.

Such best practices are closely related with project success — but what if it’s only a correlation? What if improving best practices, does not improve project success rates, because they are not the true cause of project failures?

What if the true root causes are the following less explored reasons:

Solving the wrong problem , or

, or Not finding a good enough solution to the problem,

Attempting to change systems without truly understanding how they work today and,

Allowing psychological biases to influence investment decisions

Why best practices are not the root cause

Awareness of project methodologies have been evolving for decades. The number of skilled and certified people in the industry has been growing rapidly. The number of PMI certified project managers have been growing exponentially, so have the number of architects and business analysts.

Despite the growing capabilities and skills on project teams, the failure rate hasn’t budged much since the Standish Group and others began measuring this in 1994 — therefore it is logical to conclude that improving best practices and skills are correlated to project failure rate — but are not the root cause.

Improving best practices and number of certified professionals, does not improve project success rates.

We need to go beyond the common belief and dig deeper to find more likely root causes for why innovation projects (those attempting to implementing new ideas) and change projects (those replacing existing systems) are most likely to fail.

Possible Root Causes of Project Failures

Solving the wrong problem

Organizations and teams investing in innovation don’t spend enough time defining the most important problem they’re attempting to solve. Many have considerable difficulty even identifying which problems are crucial to their missions and strategies.

Teams speed toward a solution, fearing that if they spend too much time defining the problem, their superiors will punish them for taking too long to leave the starting line.

Many times, projects go down the wrong path. They end up implementing the wrong system and discover that they didn’t solve the problem they set out to fix in the first place. They only realize in hindsight the right problem to focus on and which path to follow.

This results in shifting business requirements, missing deadlines, and re-work as teams go back to the drawing board. This eventually results in schedule and budget overruns.

The rigor with which a problem is defined is therefore the most important factor in finding a suitable solution.

Most organizations are not proficient at articulating their problems clearly and concisely. Teams need to become a lot better at asking the right questions earlier and not starting initiatives too soon. They need to become proficient at using various problem-solving techniques such as the “Five Whys” to be sure they are tackling the right problems. They also need to spend more time earlier in the project analyzing if the economic cost benefit is viable before starting them.

Not finding a good enough solution

The next step after figuring out the problem, is to develop a solution. This requires building a minimum viable product (MVP) to learn what customers want as quickly as possible, then measure, and fine-tune the product until it reaches the tipping point.

However, teams struggle to understand what constitutes a MVP and if what they have built is a good enough solution. Teams often get stuck in endless development cycles trying to meet the arbitrary standards of insiders, causing schedules to slip and budgets to balloon.

The MVP is a version which allows teams to collect the maximum amount of validated learning with the least effort. The MVP is considered good enough when customers view it as a quality product and there is diminishing returns from further improvements that customers will never notice.

Developing the MVP is a fine art of architecting a solution that balances economics, customer-development, technical debt, and extensibility. Organizations and teams need to improve their use of techniques such as design thinking, prototyping, and user development research. They need to get better at validating if what they are building is useful to customers, much earlier.

Not understanding the current state

Projects aiming to transform existing systems by modernizing or replacing older systems fail because they often skip a major step — understanding the current state, “really really” well.

Teams skip current state analysis or brush over it with the justification of starting the project faster. Current state analysis is viewed as a waste of time. Teams resist doing a current state analysis for many reasons including:

Belief that what was done in the past was inefficient, no longer useful or relevant to the new reality

Poor or lack of documentation on how the existing systems and processes work

The original people who built the system or know most about it are no longer with the organization

Data models and integrations are tricky to uncover

Skipping the current state step, often results in the new system falling short in many areas:

The new system doesn’t support existing customers, products, services as well as the old system

The new system doesn’t create the correct data

Users resist adopting the new system as it does less than the old system did

The new system doesn’t work correctly with the existing internal systems or external partners

All such issues result in cost and schedule overruns as teams have to re-analyze and re-implement the gaps. Organizations need to give teams the time and budget to do it right and teams need to get better at:

Correctly understand how the business operates today

Fully appreciate the conveniences built over time into the old system

Reverse engineer existing data models

Analyze upstream and downstream integrations

For most transformation projects, unless the business environment or business model has changed, some or all the existing capabilities will be needed in the future. The capabilities may take new form, but their essence may remain and must be well understood.

Challenges estimating enterprise IT project schedules and cost

The root causes explored so far offer reasons for why projects are late, but they don’t explain why 70% of projects are consistently over budget by approximately 40 to 50%. The average has been hovering close to 43%, since the Standish group started tracking this in 1994.

But why is the range “40% to 50%”?

When teams discover that more work is needed, the tendency is to request more funding to make it better. Funders become locked-in and entrapped in a course of action and resort to ‘throwing money’ to make the venture succeed. This is a psychological bias, known as the escalation of commitment to a course of action. Results from related psychological studies [968, Knox and Inkster[35], Staw and Fox[38] ] show that people will continue to invest approximately 40% more than they originally planned when they have a sense of personal responsibility, to avoid sunk costs, or desire not to appear wasteful.

But how far will funders go before they turn off the funding tap?

Over-funding projects will continue until the second psychological bias of loss-aversion kicks in and “the losses start to loom greater than the possible gains”. The idea of loss aversion is ingrained in the human mind and most people will feel the pain of loss twice as much as the pleasure of gains — explaining why the pain will become psychologically painful when overruns start to approach 50%.

The Ying-Yang effect between the escalation of commitment and loss aversion causes projects to overrun on average 43%. Such schedule slippage and budget overruns are often attributed to poor project management discipline, rather than to these fundamental psychological biases.

If the root cause of project cost overruns is psychological and has remained steady for decades, we can begin to develop solutions based on the reliability of this metric. The solution to project budgeting might in fact be to encourage teams to find solutions to problems by giving them constraints of 50% less budget or time, than what is available.

This solution would work as follows:

Example 1: with no constraints

A project has a budget of $100,000. The project manager assigns the entire $100,000 to the team building the system.

The team proceeds with the project and towards the end of the project, they discover they solved the wrong problem, or did not develop a good enough solution to the problem. They escalate this to the management team and present a proposal of how to improve the system and the cost to do so. The organization decides the improvements have a high likelihood of succeeding and provides an additional $50,000 in funding.

The team completes the project and the total cost of the project is $150,000 — an overrun of 150%.

Example 2: with constraints

A project has a budget of $100,000. However, in this case the project manager assigns 50% less budget (approximately $66,000) to the team building the system.

The team proceeds with the project and towards the end of the project, they discover they solved the wrong problem, or did not develop a good enough solution to the problem. They escalate this to the management team and present a proposal of how to improve the system and the cost to do so. The organization decides the improvements have a high likelihood of succeeding and provides an additional $33,000 in funding.

The team completes the project and the total cost of the project is $100,000 — the project comes in on budget.

Conclusion

Despite the increasing maturity of software development methods, and the growing number of certified professionals, software project budgets and schedules are likely to continue to be very challenging to predict accurately.

The root cause of projects over-running are human issues:

Solving the wrong problem , or

, or Not finding a good enough solution to the problem,

Attempting to change systems without truly understanding how they work today and,

Over investing because of the phycological biases of escalation of commitment and loss aversion

Projects can be delivered on time and on budget by directly countering the root causes:

Finding the right problem to solve and finding a good enough solution — using design thinking, prototyping and customer development techniques

Getting a deep understanding of the how systems work today — using current state analysis techniques

Controlling the urge to over invest. by doing cost–benefit analysis and encouraging teams to find a solution to the problem to a constraint of 50% less budget / time than what is available.