Estimating tasks with statistics and simulations using Python | by Sau Sheong | Towards Data Science

A few months ago, when the world was a very different place, I wrote about to-do list tasks strategies. I wanted to know, given a bunch of tasks in your todo list, what strategy should you use to pick the tasks to work on first? Depending on what I actually want to achieve, how I do it can be quite different.
The assumption I made there was that the complexity of a task is fixed, that is the duration it takes to complete it is pre-determined. That’s not normally true (statistics joke alert!). Complexity and therefore duration for most tasks, even before the uncertainty brought by the pandemic, is hard to predict accurately.
If we look at tasks beyond to-do lists, for example project tasks, this becomes even worse. Project task complexity doesn’t only impact duration but it also effort and costs and therefore being able to determine the complexity is pretty important.
However project tasks are notoriously difficult to estimate, and this can mean delays and escalating costs.
Berlin’s new Brandenburg airport was in planning for 15 years before starting construction in 2006 and was supposed to open in October 2011. However the project is still ongoing and is also facing a skyrocketing budget. The latest opening date is November 2020 but with the ongoing situation it seems likely to move again.
NASA’s Space Launch System (SLS), a launch rocket that will provide the foundation for manned lunar flights and a human mission to Mars started in 2010 and has cost $18.6 billion as of 2020. In early 2020 it was announced that that it will have costs overrun of $2 billion and be delayed for at least 2 years.
This nothing new though, NASA is infamous for being excessively optimistic and underestimating projects. The James Webb Space Telescope (JWST) was originally supposed to be launched in 2007 and cost $500 million, but as of today has slipped to 2021 with an estimated cost of $10 billion. Its predecessor, the Hubble Space Telescope was originally supposed to launch in 1983 but finally launched in 1990, with the costs slipping from an original $400 million to eventually $4.7 billion. The International Space Station takes the cake though — it started off with a $36.75 billion budget and it was completed six years late with final cost of $105 billion.
Construction projects are not the only type of projects that get delayed, software development projects are often the culprit as well. In a rather infamous case to develop the Queensland Health Payroll System, the project was tendered out and won by IBM in December 2007 for $98 million, and was supposed to be rolled out in July 2008. It was finally delivered in 2010, with bugs and costing $1.2 billion.
Of course there are a lot of reasons why projects fail. Perfectly well estimated projects can suffer catastrophic results for various reasons that are impossible to predict. You can be sure that however well estimated project tasks are, none could have added in the pandemic we’re facing now, nor the after-effects to it. Nonetheless, good estimations are critical to keeping a project well on track.
There are a number of ways to do estimations. In agile methodologies (mostly software development projects), we have planning pokers, affinity groups, t-shirt sizing, bucketing and so on. In terms of estimate units, there’re t-shirt sizes (S, M, L, XL etc), Fibonacci story points, dot votes and plenty of other pretty creative ways to represent the sizing of the effort.
For more traditional project management methodologies, estimation techniques include bottom-up, top-down, parametric, analogous, three-point and so on. There are plenty of other types of estimation techniques especially in software development, including COCOMO, SEER-SEM, Putnam, PERT and many others. What we definitely don’t lack are estimation techniques.
Whichever techniques and units we use, ultimately what projects are concerned about is delivering on-time (time), within cost (budget) and something that does what it’s supposed to do (quality), whether it is a space telescope or a health payroll system.
Task estimation is hard. It’s mostly judgement-based, using the experience of the estimator (usually the person doing the task). Also, most estimates tend to be overly optimistic. In fact, there is a term, coined by psychologists Daniel Kahneman and Amos Tversky in 1979, called planning fallacy that describes precisely this. What Kahneman and Tversky found was that we have a tendency to disregard what we learnt in the past about similar tasks, and also to be overly optimistic about potential issues and even our own ability to complete the task (optimism bias). The topic is pretty fascinating, if you’re keen to learn more, you can read more from their book, Thinking, Fast and Slow.
One was of getting around it is using three-point estimation. The trick is to produce 3 data points (hence the name) for every estimation:
a — the best case estimate
b — the most likely estimate
c — the worst case estimate
This forces the estimator to think through the best, worst and most likely cases each time. While this is no silver bullet to overcome the planning fallacy, it gives us some interesting data to work on.
The project evaluation and review technique (PERT) is a tool used in project management to analyse and represent tasks in a project. It was developed for the US Navy in 1957 to manage the US Navy’s Polaris nuclear submarine missle project, but found its way to many other uses including an early use to plan the 1968 Winter Olympics in Grenoble.
PERT is a tool for large scale, one-time planning and scheduling, focusing on events and timing .PERT itself gave rise to associated tools and concepts such as the work breakdown structure, critical path and critical path analysis (CPA).
PERT also gave rise to doing three point estimations, in fact it gave its name to a family of continous probability distributions called the PERT distribution.
The PERT distribution is a continuous probability distribution that is defined by the minimum (a), most likely (b) and maximum (c) values, with the mean (or expected value) of:
The PERT distribution is actually a transformed four parameter beta distribution. The beta distribution have two parameters α and β defined on the interval [0,1], making it useful for modelling probabilities and random variables. The four parameter version of the beta distribution makes the interval [a,c] instead where a is the minimum and c is the maximum value.
You can see where this is going. By using a PERT distribution to map the estimations, we can approximate the probabilities of the estimations themselves!
This is to say, while we ask the estimator to provide the optimistic, pessimistic and most likely values for the estimations, the actual expected value could be different from the most likely value, and we can figure out how likely the tasks is going to be completed within a duration or a budget.
However this is really only for a single task. In a project, of course, we have many, many tasks and each might have different distribution parameters for a, b and c. If you’re the project manager, how would you figure out the overall estimations and probabilities?
Well, we can also try to do the maths but we’re programmers, not mathematicians (I’m saying this in a good way) so we look towards our old friend, Monte Carlo.
Of course, by Monte Carlo, I meant Monte Carlo simulations and not the casino. Monte Carlo is a popular simulation technique that uses many repeated random samplings to produce numerical data. It’s like running lots and lots of experiments and figuring out insight from all that data.
Stanislaw Ulam came up with the idea in 1947 when he was recovering from an illness. He was working on a nuclear weapons project at the Los Alamos National Laboratory then, and shared the idea with John Von Neumann, who saw the significance of the idea and he wrote the first Monte Carlo simulation. Because of the secrecy of their work, the had to come up with a code name. One of their colleagues suggested, Monte Carlo, a famous casino in Monaco where Ulam’s uncle used to borrow money from relatives to gamble and the rest is history. (You might recognize Ulam and Von Neumann, both brilliant mathematicians, scientists and early pioneers of computer science from a previous article I wrote).
Monte Carlo simulations can be used to solve any probabilistic problem. Using the law of large numbers, we can approximate the results through empirical data we create. Of course, we cannot sanely do this without computers because the technique needs to run large numbers of repeated experiments. Monte Carlo literally brute forces the results!
So how exactly do we use Monte Carlo to figure out the overall estimations of a number of project tasks? It’s quite simple actually. For every task, we create a PERT distribution using the given a (pessimistic), b (most likely) and c (optimistic) values and draw a value from that distribution. What we get is a simulated estimation for each task. When we add these estimations together we get the simulated estimation of the whole project.
Rinse, repeat.
We will end up with lots and lots of project estimations, that will also be in a probability distribution of some sort. Actually, we’ll end up with another PERT distribution but let’s see how we can prove it later. Once we have the data we can figure out the various probabilities of our estimations.
Let’s give it a spin! For this I’ll be using Python3 on Jupyter Notebook, with the usual suspects — numpy, pandas, scipy and matplotlib.
We start off with a single task and run 10 million experiments on it. It’s not as crazy as it sounds, running this script just takes 2 seconds.
Let’s walk through the script. no_of_experiments and num_bins are the number of experiments we want to run, and the number of bins we use for our histogram.
tasks is a dictionary with only 1 key-value pair. I use this because naturally we’re going to have more tasks later, so just preparing ahead. The key is the name of the task, while the value is a list of 3 elements, the first is the optimistic value a, the second is the most likely value b and the last is the pessimistic value c.
Remember what I said earlier that the PERT distribution is a transformed four parameter beta distribution? I said that because neither numpy nor scipy have PERT distribution, instead they have beta distribution. So to use the PERT distribution we need to convert a, b and c to α and β. The formula for the conversions are pretty straightforward, if you don’t want to work it out yourself, you can find it from the Wikipedia entry on PERT distribution.
and
Once we have α and β we can use numpy (or scipy, take your pick but I used numpy here) to randomly draw a number of random variables from the distribution. Here we’re drawing 10 million random variables.
Next we need to convert the random variables back to the actual values we want, which is between a and c. Finally we have this extra step to add it to a points list of all 0s, which is not really useful for just a single task but when we have more tasks it will be needed. The points list is now a list of 10 million numbers randomly drawn from a PERT distribution.
Now that we have the data, we’ll plot it using matplotlib’s hist function. We change the x axis a bit to start from 0, and also change the points to represent a probability.
Run it and this is what we get.
That’s a nice curve but what does it all mean? Let’s run some statistical analysis on it using scipy.stats.
First thing we do is to simply show the tasks, and if we have more than 1 task, we add up the optimistic, most likely and pessimistic values.
Then using the stats.describe function we get the min, max, mean, variance, skewness, kurtosis and also standard deviation values. That’s a lot of stuff from just a couple of lines of code!
Finally we print out the probabilities with 0.5, 1, 2 and 3 standard deviations.
Now let’s take a look at the results.
Tasks-----        optimistic  likely  pessimisticTask 1           4       7           12Total            4       7           12Statistics----------minimum: 4.0 maximum: 12.0mean: 7.3standard deviation: 1.5skewness: 0.22kurtosis: -0.60Probabilities-------------* between 7.0 and 7.7 : 36.0% (within 0.5σ)* between 6.6 and 8.1 : 64.7% (within 1σ)* between 5.8 and 8.8 : 97.3% (within 2σ)* between 5.1 and 9.6 : 100.0% (within 3σ)
The tasks is quite self explanatory as well with min and max. The mean value is interesting, if we take the earlier equation for the mean of a PERT distribution, we get 7.333 which validates our answer. Using the variance for a PERT distribution which is:
We will get a variance of 2.222 and the standard deviation, which is the square root of variance will be 1.49 which again validates our answer.
For the probabilities, we simply take the area under the curve between the mean and multiples of the standard deviation σ on both sides of the mean, and that would be the probability (because the area under the curve will all add up to 1.0 or 100%).
It seems astonishing and wrong that we say the probability that the task will end up between 5.1 and 9.6 is 100% but that’s because I truncated the results to 1 decimal point. If we allow it more decimal points it would be something like 99.9994%. Nonetheless let’s look at a few interesting points.
The expected value or mean and the most likely value (which is provided by the estimator is close but not the same).
The skewness is 0.22, which means the estimations tend to lean towards being more optimistic. If it’s a negative number it means it tends towards being more pessimistic. Skewness of a distribution that is symmetrical is 0.
The kurtosis is -0.6 indicates that the peak is a bit flat and more spread out. Generally speaking, the lower the number (the kurtosis of a normal distribution is 0) the flatter the curve is and the higher the number the more peaky it is. A flatter curve means the the optimistic and pessimistic values are further away from each other, and the the peaky curve means the optimistic and pessimistic values are closer to each other.
There is a 99.9994% chance that the task is somewhere between 5.1 and 9.6. Now compare that with the optimistic value of 4 and the pessimistic value of 12. It seems wrong that if you think the task is optimistically a 4 and pessimistically a 12 to then be told off by statistics that either values have a probability of only 0.0006%!
Numerically it seems counterintuitive but you can interpret it from a different perspective. If the probability of the pessimistic value is only 0.0006% then maybe the pessimistic value estimation is wrong and should be larger. You can give it a quick try and change the tasks to:
tasks = {    "Task 1": [4,7, 24],}
Running it again will give us this results:
Tasks-----        optimistic  likely  pessimisticTask 1           4       7           24Total            4       7           24Statistics----------minimum: 4.0 maximum: 23.5mean: 9.3standard deviation: 3.3skewness: 0.70kurtosis: -0.02Probabilities-------------* between 8.5 and 10.2 : 33.7% (within 0.5σ)* between 7.7 and 11.0 : 65.0% (within 1σ)* between 6.0 and 12.7 : 96.3% (within 2σ)* between 4.3 and 14.3 : 99.6% (within 3σ)
You can immediately see that for the value 12 it falls within 2σ. You might be thinking — that’s not right, the statistics is telling you it’s a larger value now but how does it know?
The answer is that it doesn’t. As the estimator you know better how likely the value 12 is. If you think that it’s extremely unlikely, and there is a very very small chance that it might be correct, then the value 12 is probably correct. If you think it’s unlikely to be 12, but the chances are that it’s going to be larger than 0.0006%, then you should choose a larger value.
In other words, the numbers are what they are, but the judgement of the optimistic, most likely and pessimistic values depends on the estimator.
What we did before is just warming up. What we really want is to estimate multiple tasks. While we can more or less calculate the values for a single task due to the PERT distribution, it’s going to be pretty hairy to do so if you have 100 tasks for example.
Almost all the code is going to be the same, except that we have more tasks so I won’t reproduce it here.
tasks = {    "Task 1": [4,7, 12],    "Task 2": [2, 3, 9],    "Task 3": [3, 5, 16],    "Task 4": [6, 7, 23],    "Task 5": [10, 12, 28],}
With this list of tasks, let’s run the script again.
Now let’s do the statistical analysis again.
Tasks-----        optimistic  likely  pessimisticTask 1           4       7           12Task 2           2       3            9Task 3           3       5           16Task 4           6       7           23Task 5          10      12           28Total           25      34           88Statistics----------minimum: 26.3 maximum: 69.7mean: 41.5standard deviation: 4.8skewness: 0.40kurtosis: 0.06Probabilities-------------* between 40.3 and 42.7 : 32.2% (within 0.5σ)* between 39.1 and 43.9 : 67.6% (within 1σ)* between 36.6 and 46.3 : 96.1% (within 2σ)* between 34.2 and 48.8 : 99.7% (within 3σ)
Because we have multiple tasks now, we can’t look at the optimistic, most likely and pessimistic values for a single task. Also we need to consider if the tasks are dependent on each other. Let’s keep it simple and say the tasks are all dependent on each other, meaning Task 5 is dependent on Task 4, which dependent on Task 3 and so all until Task 1.
So logically for the project, the task estimation should be a total of the optimistic, likely and pessimistic columns. As a project collectively we see the optimistic total is 25, the most likely total is 34 and the pessimistic total is 88. But how accurate is it? Let’s dig deeper.
From the numbers, we see that the mean 41.5 is pretty far away from the most likely value, and the mean is actually the most likely value for this distribution! This means by adding up the most likely values, we end up having a more optimistic view that the numbers show.
The minimum is only slightly more than than the optimistic value while the maximum is quite a bit less than the pessimistic value. Again this means by adding up the pessimistic values, we are really being too pessimistic while we are only slightly more optimistic than the numbers show.
The skewness of the distribution is quite positive so the over project tends to be more optimistic while the kurtosis is slightly positive (actually it’s pretty close to 0) so the project is quite well balanced in terms of the range between the optimistic and pessimistic values.
Now let’s look at the probabilities. You will notice something interesting — there is around a 67% probability that the project values are within 1σ of the mean, 96% probability that it’s within 2σ and 99.7% within 3σ. Do the numbers look familiar?
If they do then congratulations, you weren’t sleeping during your statistics lessons, at least not when your lecturer was teaching you about the normal distribution. The empirical rule states that for normal distributions, 68% of the values are within 1σ of the mean, 95% are within 2σ and 99% are within 3σ. While the empirical rule is for normal distributions, most bell-shaped distributions (like the one we got) somewhat follow this rule, though with slightly different numbers (this is also known as the three sigma rule). This is also the reason why the earlier run for a single task also follows roughly the same numbers.
Now take a second look at the probabilities again. The numbers say that there is a 99.7% chance that the project value is somewhere between 34.2 and 48.8. Remember that if we add up all the values in the various columns, the optimistic value is 25, the most likely value for the project is 34 and the pessimistic value is 88! Neither the optimistic nor pessimistic values are real, and the most likely value is likely to be wrong too.
Adding up the columns doesn’t tell the real picture after all. And if you think about what it means, that’s actually what it should be. For the pessimistic values or the optimistic values to add up it means every task needs to be the all pessimistic or all optimistic, which is highly unlikely.
So what do the statistical numbers actually mean? The statistics we worked out here basically mean that if all our estimations are accurate, then the project values will be 99.7% likely to be between 34.2 and 48.8.
Project task estimations are tough and all the statistical analysis shown earlier doesn’t detract from the need to for expert judgement. Ultimately no matter how powerful the techniques are, if the estimation data going in is totally wrong, nothing will be able to make it right.
However, analysing the data through such techniques allow us to detect potential issues with the task estimations. It will also help us understand the estimations, not only from an individual basis (which can be helpful as you can see from above) but groups of them, in a project or otherwise.
The exercise we went through earlier has a simple dependency of a task to a previous one, but in a more realistic scenario, we would have tasks that have multiple dependencies, groups of tasks which are dependent on each other or even cross dependencies.
Once complexity increases it becomes much more harder to see the project as a whole and its estimation can go awry very easily. Using a statistical tool can help us to spot potential problems and guide us to better estimations.
Such statistical analysis also helps us better guide projections for costing and timeliness. As the tasks complete we also can use the analysis to compare with the actual values, then update and revise future task estimations for better accuracy.
Throughout the exercise earlier I discussed task values and project values but I was quite vague about the unit of measure itself. Was I referring to duration, was I looking at the time needed to complete the tasks and the projects, in days or weeks or months? Or was I looking at effort, in terms of man-days or man-weeks or man-months? How about cost, in dollars and cents? Or complexity?
Actually we can use this for any of the above as long as there is numerical value to whatever we’re trying to measure. As mentioned earlier, some in some agile methodologies, the teams come up with their own units of measuring the tasks (someone came up with NUTs, or Nebulous Units of Time, scrum even did away with units altogether) and that’s ok.
However we should also understand at the end of the day, project success is commonly measured on whether it is on time, within cost and with quality.
I used Python and Jupyter Notebook here to do all the work. For many project managers, that’s not a natural tool to use — most people are more comfortable with Microsoft Excel to do all these calculations.
The good news is that Excel is an excellent tool and has all the necessary functions to do the statistical calculations. The bad news is that Excel has a max of 1,048,576 rows and 16,384 columns per sheet. So if you’d like to do 10 million Monte Carlo simulations you might not be able to do it one simulation per row.
In fact we we did 10 million simulations per task and if you have a large number of tasks, you’re going to be out of luck. Also, even if thats not an issue, processing 1 million rows will take quite a while for Excel while our Python script running in Jupyter Notebook only took 2 seconds for a single task and around 7 seconds for 5 tasks.
Ulam and von Neumann were working on the Manhattan Project when they came up with the idea of Monte Carlo simulations, PERT was created to support the US Navy’s submarine nuclear missile project, Polaris. It seems ironic that both these techniques were invented due to military needs, both relating to research on nuclear weapons.
War and military ambitions have always been one of the strongest drivers of innovation in history. While these ambitions have brought misery to millions of people, it’s reassuring that out of such horror comes such brilliant tools that can also help make our lives easier.