5 Time Estimation Techniques Engineers Are Not Using
By: James Williams
Tags: Software Development, Project Management, Software Engineering, Engineering, Time Management

Fully one in six of the projects we studied was a black swan, with a cost overrun of 200%, on average, and a schedule overrun of almost 70%. This highlights the true pitfall of IT change initiatives — Harvard Business Review

This staggering number is trumped by 43% of projects that go over budget. And, let’s not stop there; only 31% of projects meet their goals. Despite a planning-heavy industry, engineers expect to fail.

As a result, projects fail a lot.

Do you find yourself struggling to make deadlines? I bet you have been on projects with detailed planning, only to find yourself lost in a sea of lateness.

Software is no stranger to good intentions with missed deadlines. Like most things, human psychology has a significant impact on why we do what we do, including planning.

In this article, I cover tried and true ways to estimate time and meet your project goals.

But first, let’s look at why existentially.

The Planning Fallacy

Way back in 1979, Daniel Kahneman proposed the planning fallacy along with Amos Tversky. In short, because of optimism bias, we are likely to underestimate time. As humans, we don’t do a great job of using past experiences to predict future tasks.

Does this sound familiar to you?

It’s not surprising that many of the techniques to plan a software development project are optimistic. “As an engineer, we want the ability to deliver a project on time.” And, in the voice of the Agile Manifesto, we want to get things done-done with transparency.

While this approach can work, it fails to handle what-if scenarios properly.

If you have been on grossly optimistic projects, I have some helpful tips.

The online shopping cart that failed

I was overseeing a project for one of my teams, who were great at building online shopping experiences. The team was packed full of talent, and they were willing learners.

Their goal was simple — build a shopping cart that allowed users to login with their Facebook credentials. These ‘A’ players made a shopping cart before leveraging Google credentials. I couldn’t have picked a better team.

They started with a bottom-up approach littered with detail.

As a user, I would like to log on with my Facebook credentials

As a user, I want to click a ‘Forgot Password’ link to retrieve my lost password

As a site administrator, I want to be able to see the number of active users.

You know the drill.

I asked a straightforward question, “Do you have a plan if something goes wrong and blows up in your face? Usually, these simple questions are followed up with a lecture on user personas, ordered steps, the definition of done, and the bullet proof user stories the team developed.

With great confidence, somebody told me, “it should be simple.” The look on one of the engineer’s face was, here goes this old dude again with his old-school techniques and questions.

I let them proceed:)

After all the bottom-up planning and agile folklore, the estimate was 6–8 weeks. I pointed out that a two-week delta was a 30% buffer. I was blazingly told, “Well, let’s go with eight weeks; we got this!”

Six weeks later, they were still trying to get the Facebook login to work. It took everything in me not to say, “Didn’t you tell me, you got this?”

But, I digress.

Failure amongst software projects are rampant and expected. Apathy is typical, and most engineers shrug their shoulders hopelessly.

Why?

Estimating the time it takes to build software is beyond art. It is truly one of the hardest things to get right. Most college programs don’t teach you how to manage time, and the hoards of recommended tools that are supposed to help, don’t help. They are simply roadblocks and, in many cases, cost more time than they save.

Here are the five techniques I’ve used in my career that will help you deliver on time.

1. Work With An Authority Partner

By Retha Ferguson from Pexels

An authority partner is a trusted person that you have a discussion with about strategy. The first thing I do on new projects is to pick someone I trust and have a conversation. There is no computer needed and not even a whiteboard. I use good old-fashion words to describe the overall objectives. And if I am lucky, I have chosen an authority partner willing to poke holes in my theories right from the beginning.

Over the years, planning projects have become computational. That is, you sit down and break down your tasks. You use some system to assign time or complexity, and you sum it all up at the end.

This computational method fails, because there is no context.

A corollary I like to follow in chess that says, “Evaluation is superior to calculation.” Rather than calculating 20 moves deep, you take a step back and assess the board and come up with a plan. How do you study chess? You get a partner who knows what they are talking about and discusses strategies.

If you want to become an expert and make your deadlines, spend time discussing “what” you are doing and “why.” It’s essential to discuss your plan before you start the activity of building tasks lists. The Agile Methodology has done a great job of forcing us to think about the user, but it’s not a strategy; it’s a mechanism to do something.

2. Intentional IF-THEN Planning

The most frequent question asked when things go wrong is, why didn’t anyone think of that? Customers are using what you’ve built, and you are sitting in a post-mortem meeting trying to sort out what happened. After all, you did plan.

But while planning is essential and the planning mechanisms we use, it’s superior to think if-then scenarios. “If-then” scenarios are a way of planning, that upfront cites what you will do if plan ‘A’ doesn’t work.

Let’s say one of your projects use AWS’s RDS to load a database schema. This is important because you are migrating from a self-hosted database, and the goal is to move the data. What should your user stories say? How granular should your tasks be? What questions should you ask yourself?

Here are some “if-then” scenarios you want to add above beyond your user stories:

If AWS’s RDS doesn’t work in the next hour, then I will revert to the old database? Do I have a plan for this?

AWS’s RDS doesn’t work in the next hour, I will revert to the old database? Do I have a plan for this? If for some reason, I can’t get all the data to copy by 6 PM, then I will set up partial database scripts to copy a portion of the data so that I can show something is working.

for some reason, I can’t get all the data to copy by 6 PM, I will set up partial database scripts to copy a portion of the data so that I can show something is working. If I can’t get the RDS credentials working in the next 30 minutes, then I will reach out to someone for help.

Notice that each bullet is intentional. It’s essential to plan what, where, when, and how you will do something, not just now long.

To state it another way, you need to plan how you will achieve a task buried in a user story.

“Everyone has a plan ‘till they get punched in the mouth.” — Mike Tyson

3. Plan Backwards

By Charlie Solorzano from Pexels

Backward planning is a technique used to counteract the planning fallacy. It helps you work from an end-state backward. While you can use this method on large scale projects, it also helps to use this technique on smaller tasks.

The trick is to start from the end and plan towards the beginning.

Here is a framework of questions you can use to work backward.

What’s the delivery date?

How long does it take to push code to production?

How long do integration tests take? What about ad hoc testing?

How long do regression tests take?

What day can you schedule a code review? How long do code reviews take? How long will it take you to make changes if needed?

Are there any special circumstances before submitting your code?

How many lines of code do you have to write? What’s been your average?

How many meetings will you have? With Whom?

A pro tip is to put each of the dates on your calendar.

Contingency Planning

While backward planning is proven to be highly effective, there is still more work to do. Even the best backward plan is met with circumstances that are out of your control. A good example is when a business moves a timeline up, or an extra feature added.

How do you handle these?

Plan for it. Here is a sample “game script” I’ve used on projects in the past.

If the delivery date is moved up by 25%, drop a feature

If the delivery date is moved up by 50%, drop two features

If I have to attend to a production bug that takes more than one day of my time, drop a feature

A bonus tip is to write this in an email and send it to stakeholders to inform them of the costs upfront. You will be surprised and how many people try and help you.

4. Connect Tasks To Outcomes

By Anna Shvets from Pexels

A great offender of underestimating is the forgotten task. Most project planning is an exercise in trying to think of every task, adding up the time it takes to do each one, providing a buffer of say 20%, and then computing your total. I’ve seen 50% buffers, resulting in a project time four times as long as planned.

The problem with this approach is context is lost. There’s no “why,” no metric, and engineers focus on the wrong objective. Standing up a web server as part of a project is a terrible goal, and it has no metric for success.

That’s right, the one you thought was small and overlooked takes longer than the code you planned to write. You end up with lots of completed, one-of tasks, yet an incomplete and late project.

Robotic task-based thinking is senseless, and we know that it doesn’t work. Ask yourself this, what is a task? How big should a task be? How much time should a job take?

Far too often, as all tasks are not created equal, the system is implicitly flawed.

Here are a few example tasks that every engineer forgets to consider:

The time it takes to do a simple git-clone and to get a build working

Updating Node.js to a newer version — this applies to almost every language, not just Node

If you are switching between projects, how long did it take you last time to get your environment setup?

Time to read the documentation for a framework or library you are using

Breaks!

Reserve time to design something — literally time to pseudo-code and draw diagrams

Time to fail

Not surprisingly, a list of infinite tasks is a recipe for failure and is impossible to manage. This approach is rigid and commits you to a specific plan that fails most of the time.

The key to being any good at managing your time is to commit yourself to outcomes ruthlessly. Here are a few examples of ways to think about results rather than solely tasks.

5. Add Top-Down Strategy To Your Bottom-Up Plan

Instinctively, engineers estimates from the bottom up. You break down tasks, use some estimation system, and develop what they believe to be accurate.

Task one takes 4 hours, then task two takes one day, and so on.

Here’s a little secret, bottom-up planning usually fails. Why? Because bottom-up planning worships the task and not the result.

A better approach is to think of a strategy first and define an outcome. What is the expected value delivered? What happened last time on similar projects, and what were your learnings? What are the metrics for success? What is the outcome expected by your users?

My grandma would say, “Planning how long it takes to drive somewhere works until you discover you need a boat on your trip.”

Goals are superior to bottom-up tasks

Here is a super simple example. If you define a goal to build a payment gateway, a good business outcome is a user can use their credit card to make X amount payments. This top-down strategy is clear to you and easy to explain.

A Top-Down approach admits that it’s the outcome, not the tasks, that matter.

And, if you focus on the outcome, you have a better chance of altering your “not so perfect plan” when things go wrong.