Estimating AI Project Costs & Timescales: 4 Rules of Thumb
By: Was Rahman
Tags: Machine Learning, Project Management, Artificial Intelligence, AI, Data Science

But, first a reminder of some of the basics of IT project estimating. Estimating AI project costs is an evolution and adaptation from that.

The Fundamentals of Reliable IT Project Estimating

IT work has an often-deserved reputation for massive over-runs and overspends, especially large, multi-year projects. However, over the last couple of decades, the reality has changed. IT projects still run late, but huge, eye-catching failures are far less common.

There have been several reasons for this change. One is the prevalence of more flexible development processes and shorter project cycles. Another is the adoption of project quality metrics and continuous improvement. Carnegie Mellon’s CMM work was instrumental in the second and is still invaluable reading for IT newcomers.

There is still heated debate about IT processes and project management methods. However, a request for IT project quotes generates broadly comparable estimates from most vendors, given the same information. This will include similar breakdowns of how the overall work will split into phases or activities e.g. requirements etc.

This is relevant for AI projects because we’re currently where general IT estimating was many years ago: inconsistent, often unreliable. A big cause is that AI development processes vary, as do opinions on what quality means for AI deliverables. So discussions about estimates are typically driven by personal experience or project constraints rather than something more objective.

Creating consistency in how you build AI will result in a more objective basis for estimating future AI projects. This will also allow you to improve and optimise your AI work, and hold your own in vendor discussions.

Estimating AI Projects: Rule of Thumb 1

Don’t Plan Big Projects Without Credible Benchmarks, Ideally Your Own

No matter what you’re promised or the figures indicate, you should embark on big AI projects judiciously. If you’re experienced in AI work, you probably have some benchmarks on what’s involved. These are the best way of validating large predicted AI project costs. Even so, it’s worth asking for other objective data to support estimates, ideally ones you can validate. Client references are particularly helpful.

If you’re experiencing your first taste of AI, the risks of starting with a large project are greater. I’d invariably suggest starting with a short, relatively low-cost exercise.

With that, your teams and partners will have experience of AI in your organisation. This will be a real experience of developing AI solutions that use your data and connect to your IT systems. This is a more realistic basis for future estimates of AI project costs and timescales than generic or industry figures. After a second or third small project, you’ll have a more reliable starting point for estimating more ambitious AI work.

When starting out with AI, useful small projects can be as short as a 2–3 months, rather than quarters or years. Realistic results are possible with tens or very small hundreds of thousands of pounds/dollars/euros; costs in the hundreds of thousands or millions create much greater pressure on results. And a core team of single-digit size can be more effective and productive than one of dozens.

If people advising you say otherwise, then you may want to look again at the choice of scope or ambition. And of course, there may be organisational or commercial factors influencing views.

Estimating AI Projects: Rule of Thumb 2

Accept Some Phases are “Experimental” in Nature, But Keep Them On a Leash

A sometimes misunderstood feature of AI work is that it’s normal for days’ or weeks’ work to be “wasted”. This isn’t necessarily a sign of team inexperience or lack of competence — although of course productivity improves with practice. It’s a reflection of underlying AI techniques, which require experimentation and a degree of trial and error.

A challenge in estimating AI project costs & timescales is they can become — perhaps inadvertently — “blank cheques”, especially for AI vendors. The trick is finding the right balance between enough latitude for effective results, and work that’s allowed to “drift”.

Most AI courses, including non-technical ones, refer to the concept of EDA (or some equivalent). This is Exploratory Data Analysis, a crucial early part of most AI projects.

Tips for Handling Estimates for “Experimental” Work

One common approach is to separate estimates for AI project costs and timescales into two parts. A firm estimate is possible for EDA, with a more tentative one for the rest of the project. The second estimate is only confirmed at the end of EDA. By then, there’s a better understanding of the data, business problem and wider IT environment.

Another option is to use IT development techniques like timeboxing to limit the duration of the experimental work. This, in turn, constrains the rest of the project to what was discovered in the timebox. This touches on a discussion of development methods like Agile. It’s not appropriate to get into the details of that here, other than to observe that Agile isn’t an excuse for poor estimating.

How you approach the “experimental” aspects of an AI project is less important than recognising their characteristics. For good AI project estimating the trick is to limit the duration of experimental phases, without excessively compromising their usefulness.

Ideally, by using such phases well, your team discovers unavoidable errors or dead-ends after days or weeks, not months.

Estimating AI Projects: Rule of Thumb 3

Decide How You’ll Balance the Tension Between “Good” and “Good Enough”

A key role of data scientists is exploring different ways of solving business problems, evaluating alternatives to identify the best. Whereas business members of an AI team is to maintain focus on the business results of AI work. From that perspective, AI is a means to an end, to be reached as quickly and cost-effectively as possible.

This can lead to two viewpoints on what makes a task “complete”, or needing more effort. For example, if an algorithm hits 95% “accuracy”, data scientists may consider this poor, but business users may believe otherwise.

The right answer, of course, depends on what the algorithm is being used for. This includes understanding the value of getting an answer 95% “right”, and the cost/risk of 5% inaccuracy. This is where terms like Precision, Recall and F1 score become relevant and lead to discussions such as the relative importance of false positives vs false negatives.

What Does This Balance Look Like in Practice?

To understand this a little more, consider an example from medical diagnosis, say cancer detection. The meaning of “5% inaccuracy” is ambiguous. One possibility is that 5% of the patients screened believe they don’t have cancer, but actually do. Conversely, it may mean 5% of patients flagged as having cancer are actually all clear.

The decision to pursue a more accurate algorithm or stop at 95% accuracy needs understanding of such differences. Ideally, there’s also an awareness of the associated business value.

For more accuracy, further work tuning the model may be the answer;

If 95% is approaching the limit of how well the algorithm can perform, other algorithms may be required;

A third option might be revisiting options to source or prepare better data;

The right answer may be to accept 95% accuracy, even if better is possible.

This is just one example. There can be several such potential conflicts on the “right” answer during AI work. Each will impact AI project estimates.

A role of AI project leadership is to balance these two mindsets. It’s tricky enough during the project. Even with facts, there may have compelling arguments for both viewpoints.

It’s difficult — arguably impossible — to accurately allow for such possibilities before they arise. So when estimating AI project costs and durations at the outset, allowance needs to be made for this. This includes agreeing within the team how to handle such decisions.

Estimating AI Projects: Rule of Thumb 4

Have Realistic Expectations & Contingencies for Data “Gruntwork”

The fourth rule of thumb arises because AI results usually require substantial data preparation work. This work is often under-emphasised, and sometimes overlooked, in project estimates.

Getting enough appropriate data to “feed” an AI solution can be a black hole in AI work, especially for the unwary. Data preparation can suck up inordinate amounts of effort, and sometimes may not even be part of vendor quotes.

One challenge is knowing how much data gruntwork is really needed for your circumstances. The other is making sure it’s been included appropriately in estimates.

Look especially carefully at vendor estimates for issues here. Data preparation can sometimes be missing from quotes completely, buried in proposal assumptions. Another approach is to allow a token few days, knowing full well it will expand immediately.

Estimating Data Preparation Work

To cover such possibilities, your team needs to understand what relevant data is available. This includes how complete, accurate and reliable it is, and what may and may not be done with it (e.g. legal, regulatory, ethical).

Without this understanding, it’s unlikely anyone can reliably predict effort and costs of preparing data. This understanding includes what data there really is, what state it’s in (completeness, accuracy etc), and the work to prepare and clean it. It’s another tricky part of estimating AI project costs and timescales.

If your organisation has done AI or analytics work in the area you’re considering, this information may be available. It may not be in formal documents, but hopefully, there will be people with relevant knowledge to help size the data preparation work.

If you don’t have the knowledge available to understand data preparation, you’ll need a scoping exercise, called Exploratory Data Analysis (EDA). If vendors do this, you should expect clarity on data preparation activities and estimates as a deliverable.

Estimating AI Project Costs & Timescales: Wrap-up

AI Processes & Benchmarks are Key to Long Term Estimating Accuracy

Estimating AI project costs and timescales is harder than for regular IT work. In part, this is due to the lack of AI industry processes and benchmarks. The result is more reliance than ideal on the experience, judgement and opinions of your AI experts and vendors.

If AI is something you’ll do often, consider creating your own approach to AI work over your first few projects. You’ll be capturing AI project cost and effort data for your own organisation. This will be more reliable for you than industry figures, and provides benchmarks for continuous improvement. It isn’t necessarily a difficult exercise and doesn’t add much overhead. In fact, it should pay for itself quickly through savings in future work.

Estimating Smaller Projects is Easier, Especially Early On

In the meantime, starting small is usually the way to go when getting started with AI projects.

Regardless of project size, there are some obvious areas where estimates may be excessive or over-optimistic. This is especially the case if you’re relying on vendors to deliver your work.

Consider Estimating Each Phase As You Go

Effective use of EDA can be very helpful, along with good project management practices — especially iterative approaches like Agile. You’ll also need to balance the instincts and priorities of different team roles, especially during “experimental” phases of work.

This article was first published on www.aiprescience.com .