The “Bias-Variance Trade-Off” Explained Practically (In Python) | by Samuele Mazzanti | Towards Data Science

The “bias-variance trade-off” is one of the most frequent topics in data science interviews. Still, many candidates struggle to understand the concept in depth. I guess it happens because this topic is always explained from a fully theoretical point of view.
However, I believe that the best way to understand something is to do it yourself — or, better, to code it yourself.
You don’t really understand it, until you can’t code it!
In this article, with the aid of some data, we will see what the bias-variance trade-off means in practice, and how to compute it in Python (both from scratch and using an out-of-the-box implementation).
The usual definition of the Bias-Variance decomposition is:
where MSE stands for Mean Squared Error and θ represent the parameters of the model (for instance, in linear regression, θ would be the vector containing all the regression coefficients).
But there is a problem: we can never observe the true value of θ. Moreover, in some models, it’s not possible to find explicitly what θ is. So this definition is quite useless from a practical point of view.
However, what we observe in real life is the ground truth, which is the realization of the target variable on some test data (often called y). So, from our perspective, it makes much more sense to replace θ with y, and obtain the following equation:
where:
This formula is much more convenient because in a real-life setting we actually know all these quantities.
Also, with this version of the formula, we are able to give a more informal interpretation of MSE, Variance and Bias:
This decomposition is often used to explain how the outcome of a model changes based on its “flexibility”.
Low flexibility. All the estimated models tend to be similar, thus Variance is small. On the other hand, the “average model” is not powerful enough to get close to the ground truth, thus Bias is large.
High flexibility. Each model depends heavily on the specific observations it has been trained on, so the models are very different from each other, thus Variance is large. However, in the long run, flexibility allows you to take into account all the small details. As a consequence, averaging all the models allow us to get a very precise “average model”, thus Bias is small.
This is the essence of the trade-off: if the complexity is too low or too high we will have a high Mean Squared Error respectively due to Bias or Variance. The point is how to get model flexibility just right, in order to minimize both Bias and Variance at the same time.
At this point, you are probably wondering what it means to get different models (and an “average model”).
Suppose you pick an algorithm (for example Scikit-learn’s DecisionTreeRegressor) and set its hyperparameters like this:
from sklearn.tree import DecisionTreeRegressoralgo = DecisionTreeRegressor(min_samples_leaf = 10)
At this point, you can obtain infinite possible models, based on the specific training dataset on which you train the algorithm.
Suppose you could draw many different training datasets, and train a different model for each of them. Then, you could take the average of the predictions made by each model: this is what we call the “average model”.
import pandas as pd# initialize dataframe for storing predictions on test datapreds_test = pd.DataFrame(index = y_test.index)# for each model: draw training dataset, fit model on training dataset and make predictions on test datafor i in range(1, n_models + 1):  X_train, y_train = draw_training_dataset()  model = algo.fit(X_train, y_train)  preds_test[f'Model {i}'] = model.predict(X_test)# calculate "average model"'s predictionsmean_pred_test = preds_test.mean(axis = 1)
Now that we have all the quantities that we need, we can finally compute Mean Squared Error, (squared) Bias and Variance.
from sklearn.metrics import mean_squared_errormse = preds_test.apply(lambda pred_test: mean_squared_error(y_test, pred_test)).mean()bias_squared = mean_squared_error(y_test, mean_pred_test)variance = preds_test.apply(lambda pred_test: mean_squared_error(mean_pred_test, pred_test)).mean()
This is a conceptual sketch of the process we have just followed:
Let’s see an output example for 3 models and 5 test observations:
But there’s a problem.
In the code above, we have used a fictitious function called draw_training_dataset(). However, in real life, we cannot actually draw training datasets from an infinite sample space. In fact, we usually have just one training dataset. How to make up?
The trick is to bootstrap (i.e. to randomly draw with replacement) rows from our training dataset. So the process becomes:
Now we are ready to wrap everything together and write a Python function for computing bias and variance of an estimator. All we need to do is to take the code snippets above and replace the function draw_training_dataset() with the bootstrap procedure.
This is a solution:
import numpy as npimport pandas as pdfrom sklearn.metrics import mean_squared_errordef bias_variance_estimate(  estimator,   X_train,   y_train,   X_test,   y_test,   bootstrap_rounds = 100):  # initialize dataframe for storing predictions on test data  preds_test = pd.DataFrame(index = y_test.index)  # for each round: draw bootstrap indices, train model on bootstrap data and make predictions on test data  for r in range(bootstrap_rounds):    boot = np.random.randint(len(y_train), size = len(y_train))    preds_test[f'Model {r}'] = estimator.fit(X_train.iloc[boot, :], y_train.iloc[boot]).predict(X_test)          # calculate "average model"'s predictions  mean_pred_test = preds_test.mean(axis = 1)  # compute and return: mse, squared bias and variance  mse = preds_test.apply(lambda pred_test: mean_squared_error(y_test, pred_test)).mean()  bias_squared = mean_squared_error(y_test, mean_pred_test)  variance = preds_test.apply(lambda pred_test: mean_squared_error(mean_pred_test, pred_test)).mean()      return mse, bias_squared, variance
In order to see the bias-variance decomposition at work, let’s use it on some real data: the house prices dataset from Kaggle. The dataset is made of 79 predictors (such as building class, general zoning classification, linear feet of street connected to property, lot size in square feet, and so on), and the objective is to predict the final sale price (which is in thousands of dollars).
Let’s take a predictive algorithm, for instance Scikit-learn’s DecisionTreeRegressor and see how Bias and Variance change according to model’s flexibility.
But what does model’s flexibility mean? The answer depends on the specific algorithm and its hyperparameters.
In the case of DecisionTreeRegressor, we can take, for instance, min_samples_leaf. This hyperparameter determines the minimum number of samples that can end up in any terminal leaf of the decision tree. Thus:
When min_samples_leaf is high, it means that we will have few terminal leaves, each containing many samples. Therefore, the model is not flexible, because it is forced to put many different samples together.
When min_samples_leaf is low, it means that the tree is very deep. The model is then very flexible, because it is allowed to make a different prediction for any few samples.
So the idea is to make an estimate of Bias and Variance for different choices of min_samples_leaf.
We could use the function we have created previously. Alternatively, we can use the function bias_variance_decomp from the library mlxtend (here you can find a beautifully written documentation).
import pandas as pdfrom mlxtend.evaluate import bias_variance_decompout = pd.DataFrame(columns = ['MSE', 'Bias^2', 'Variance'])for min_samples_leaf in list(range(1, 11)) + list(range(15, 105, 5)):  model = DecisionTreeRegressor(min_samples_leaf = min_samples_leaf)      mse, bias, variance = bias_variance_decomp(    model,     X_train.to_numpy(), y_train.to_numpy(),    X_test.to_numpy(), y_test.to_numpy(),     loss = 'mse'  )      out.loc[min_samples_leaf, 'Bias^2'] = bias  out.loc[min_samples_leaf, 'Variance'] = variance  out.loc[min_samples_leaf, 'MSE'] = mse
This is the outcome:
It’s exactly what we expected: when the flexibility of the model increases (i.e. when min_samples_leaf decreases), Bias tends to reduce, but Variance tends to grow. This is the essence of the trade-off. Finding the right balance between the two extremes is the mission of any data scientist.
Thank you for reading! I hope this walkthrough has helped you to understand the bias-variance trade-off in depth.
I appreciate feedback and constructive criticism. If you want to talk about this article or other related topics, you can text me at my Linkedin contact.