Predicting the task duration based on a range | by Michael Larionov, PhD | Towards Data Science

In one of the previous entries we have established a statistical model for predicting the actual project time and cost based on the estimates. We discussed that we can fit the estimates (both for the Agile and Waterfall projects) to a Log-Normal distribution, which guarantees the positive support. Using statistical approach to estimation allows us to give prediction with a required confidence level, and also project monetary benefits, costs and risk, as we discussed in another post.
One thing I was asked is how the model generalizes for the case when an estimate is given as a range. Indeed, this is what everybody taught us: do not give a single number, but range. One approach is to continue to use our statistical model, and feed it a number in the middle, the mean of the two values.
That way the model can be used without modifications.
There are two problems with this approach:
Taking a mean of high and low is arbitrary. It reduces the information given by half. It would be better to have an algorithm learn where we need to set the variable x within the interval between low and high boundariesBy giving us a range of data, a developer is trying to convey to us a very important information: a degree of uncertainty in the estimates. A correct model should use that information.
Taking a mean of high and low is arbitrary. It reduces the information given by half. It would be better to have an algorithm learn where we need to set the variable x within the interval between low and high boundaries
By giving us a range of data, a developer is trying to convey to us a very important information: a degree of uncertainty in the estimates. A correct model should use that information.
To simplify the process, we will take natural logarithm of all the estimates and the actuals. Since we model estimates using log-normal distribution, our new variables y, l, h will be logarithms of the actual number of days, low and high estimates respectively. In this case we can use Normal distribution! We will model y using linear regression:
In case where θh and θl are equal, we get exactly the same problem as we discussed earlier.
The likelihood function for a single piece of data in this case can be written as follows (following this).
As mentioned earlier, by giving a range, the developer wanted to communicate to us the uncertainty of the estimate. We should include this uncertainty in our estimate of σ. Intuitively the range is proportional to the standard deviation, and we can learn the coefficient by modeling σ as:
If we also use precision parameter τ in place of σ0:
Then our likelihood function will be:
The priors for τ and θ are traditionally Gamma and Normal distribution respectively:
Here α , β, λ are hyperparameters.
The choice of prior for ζ is more difficult. None of the conjugate priors exist for the kind of likelihood function we have chosen. For now we can select the normal distribution. Zero mean of this distribution means that a priori we don’t trust the ranges (we know that many consultants the range is always 20% and does not convey any information). High mean of the prior distribution means that we pay more attention to the estimated degree of uncertainty.
For simplicity we set the mean to zero.
The negative log-posterior function is:
In this blog I will find parameters, corresponding to the maximum posterior. And to avoid making errors in differentiating, we will use TensorFlow. We will follow this example to build our code
import numpy as npimport pandas as pdimport tensorflow as tf
The data here represent the estimated and actual number of days. We see that the developer liked to add 25% to his estimate as a buffer.However for some of the stories he added more buffer, perhaps, to indicate more uncertainty.
seed=1389tf.reset_default_graph()task_data = pd.DataFrame({'low':[4,14,4,3,4,3,4,9,6,27,20,23,11],                          'high':[5,18,5,4,5,7,5,10,8,30,25,29,14],                          'actual':[17,8,5,3,5,4,9,9,4,27,16,15,7,]})%matplotlib inlineimport matplotlib.pyplot as pltimport seaborn as snsfig, ax = plt.subplots(figsize=(11.7, 8.27))task_data['story_id'] = task_data.indexdata_for_plot = pd.melt(task_data, id_vars="story_id", var_name="type", value_name="days")task_data.drop(columns=['story_id'], inplace=True)sns.barplot(x='story_id', y='days', hue='type', data=data_for_plot,ax=ax);
When defining variables, we substitute τ for another variable ρ:
This is to avoid the optimizer selecting negative τ
#Taking the log of datalog_data = np.log(task_data.values)N = log_data.shape[0]#Defining variablestheta_h = tf.Variable(name='theta_h', initial_value=0.5)theta_l = tf.Variable(name='theta_l', initial_value=0.5)zeta = tf.Variable(name='zeta', initial_value=0.01)rho = tf.Variable(name='rho', initial_value=0.01)
Since we don’t want to tune too many hyperparameters, we will set α and β to one. Both λ parameters act as regularization parameters, so we will have to tune them
#Set the hyperparametersalpha = tf.constant(name='alpha', value=1.0)beta = tf.constant(name='beta', value=1.0)lambda1 = tf.constant(name='lambda1', value=1e-4)lambda2 = tf.constant(name='lambda2', value=1e-4)def loss(l, h, y):    return tf.log(1+zeta**2*(h-l)) + \        rho**2/2/(1+zeta**2*(h-l))**2 * (y - theta_l*l - theta_h*h)**2cummulative_loss = tf.reduce_sum(list(np.apply_along_axis(lambda x: loss(*x), axis=1, arr=log_data )))cost = cummulative_loss - (N+1-2*alpha)/2*tf.log(rho**2) + beta*rho**2 + \rho**2*lambda1/2*(theta_h**2+theta_l**2) + rho**2*lambda2/2*zeta**2learning_rate = 1e-4optimizer = tf.train.AdamOptimizer(learning_rate)train_op = optimizer.minimize(cost)import mathinit = tf.global_variables_initializer()n_epochs = int(1e5)with tf.Session() as sess:    sess.run(init)    for epoch in range(n_epochs):        if epoch % 1e4 == 0:            print("Epoch", epoch, "Cost =", cost.eval())            print(f'Parameters: {theta_l.eval()}, {theta_h.eval()}, {rho.eval()}, {zeta.eval()}')        sess.run(train_op)    best_theta_l = theta_l.eval()    best_theta_h = theta_h.eval()    best_sigma = 1/math.sqrt(rho.eval())Epoch 0 Cost = 55.26268Parameters: 0.5, 0.5, 0.009999999776482582, 0.009999999776482582Epoch 10000 Cost = 6.5892615Parameters: 0.24855799973011017, 0.6630115509033203, 0.6332486271858215, 1.1534561276317736e-35Epoch 20000 Cost = 1.39517Parameters: 0.2485545128583908, 0.6630078554153442, 1.3754394054412842, 1.1534561276317736e-35Epoch 30000 Cost = 1.3396643Parameters: 0.24855604767799377, 0.6630094647407532, 1.4745615720748901, 1.1534561276317736e-35Epoch 40000 Cost = 1.3396641Parameters: 0.24855272471904755, 0.6630063056945801, 1.4745622873306274, 1.1534561276317736e-35Epoch 50000 Cost = 1.3396646Parameters: 0.2485586702823639, 0.6630119681358337, 1.4745632410049438, 1.1534561276317736e-35Epoch 60000 Cost = 1.3396648Parameters: 0.2485581487417221, 0.6630115509033203, 1.4745649099349976, 1.1534561276317736e-35Epoch 70000 Cost = 1.3396643Parameters: 0.2485586702823639, 0.6630122065544128, 1.4745644330978394, 1.1534561276317736e-35Epoch 80000 Cost = 1.3396643Parameters: 0.24855820834636688, 0.6630116701126099, 1.4745631217956543, 1.1534561276317736e-35Epoch 90000 Cost = 1.3396646Parameters: 0.248562291264534, 0.663015604019165, 1.474563717842102, 1.1534561276317736e-35
What is interesting here is that ζ is zero. This means that we cannot trust the estimation of uncertainty that the developers give us. This also means that we can just use log-normal distribution around the mean specified by the learned parameters θl and θh. Let’s say, the same developer estimated a new task to take 10–15 days. Plugging it into the formulas we see:
mu = best_theta_l*math.log(10)+best_theta_h*math.log(15)most_likely_prediction = math.exp(mu)    most_likely_prediction10.67385532327305
We can also get the 95% confidence, by plugging the values directly into log-normal distribution:
from scipy.stats import lognormdistribution = lognorm(s=best_sigma, scale=most_likely_prediction, loc=0)print(f'95% confidence: {distribution.ppf(0.95)}')95% confidence: 41.3614192940211
As we see, if we want 95% of confidence, we have to give an estimate of 41 days, instead of 11 days for 50% confidence. This is very easily explained if you see that in the past the developer did not do a very good job estimating the tasks.
You can access the notebook from github.