Bias-variance tradeoff in machine learning: an intuition | by Mahbubul Alam | Towards Data Science

If we had more money in our pockets, we tend to spend more — that’s almost a fact that everyone knows. But what’s often not known is the exact relationship between income and expenditure, i.e. how much people would spend on a known income.
An approximate solution is to build a statistical model by observing people’s income and expenditure. The more data there is, the better the model. We can then take this model and apply it to an unknown place or population with reasonable confidence.
But the model wouldn’t be able to make a 100% accurate prediction, because people’s behavior changes over time and space (well, that’s why this is called a statistical model rather than a deterministic model). However, even if the model is not universal, we want to make it sufficiently generalizable — an “average model” that is resilient to a wide range of variations and situations.
Let’s think of a different context, when we actually do want to build a model for a homogenous group of population, whose behavior is more-or-less predictable. In that case, we’ll aim for a “better-than-average model” that is suitable for that particular population. However, we know that this model is such customized for the target population that it will fall flat in an exotic context.
The average model we built is biased, but it has high flexibility and works in a variety of situations, whereas, the better-than-average model is customized, less biased but will perform poorly outside the population it was trained on.
Now let’s take both these models to the extremes. In one spectrum we simplified the model so much that it lost all its predictive power; a random guess performs better than the model. On the other extreme, let’s build the model based on every individual in a population and tune the model so much that it has high predictive power for that population.
So which model is better?
Depends on how do you want to use the model. A bad model performs badly everywhere, an average model performs on par in most circumstances, whereas a fine-tuned model performs great in one situation but not so good in others.
The purpose of building a model is to explain real-world phenomena and we use models because those phenomena can change in space and time. But can we build a model that is applicable everywhere and at every point in time?
The answer is no. And that’s where the concept of bias-variance tradeoff is useful.
Let’s now connect this intuition with the formal concept of bias-variance tradeoff.
In machine learning, each model is specified with a number of parameters that determine model performance. A good model performs well both in training and out-of-sample data.
Some models can be used out-of-the-box with default parameters. But if we do not tune the parameters, there is always the possibility that the model will not perform at its best. This situation is known as underfitting.
So we tune the parameters to fit the training data and then evaluate model performance against testing data. This process is repeated until the model reaches the desired performance. But how will this customized model perform if released into the wild? It is highly likely that this highly tuned model performs poorly in out-of-sample data — a problem known as overfitting.
Underfitting and overfitting are bad in a similar way — they are not good at making general predictions. So the bias-variance tradeoff is nothing but maintaining the balance between under and overfitting, choosing between high variance and high bias. Data scientists find a sweet spot so they are confident that the model will perform well in out-of-sample data.
Let’s now extend the intuition and the concept of bias-variance tradeoff with an example in Python. For that, I will first import a couple of libraries (numpy and seaborn) and create two small arrays — x and y — to demonstrate.
import numpy as npimport matplotlib.pyplot as pltimport seaborn as snsx = [0, 1, 2, 3, 4, 5, 6]y = [0, 1, 2, 14, 18, 32, 52]
Now let’s plot the x and y variables and fit a linear regression line.
# linear fitsns.regplot(x,y, ci=False);
The linear model does not seem to fit the data very well, it clearly looks like a bad model with most data points far from the line. It is highly biased with very little variance. So clearly there is an opportunity to improve this model.
Now let’s fit a high-order polynomial line and see how the model performs.
# polynomial fitpoly_fit = np.poly1d(np.polyfit(x,y, 6))plt.plot(x, poly_fit(x), linestyle='-')plt.scatter(x, y);
How amazing! A 6-degree polynomial fits the data perfectly — actually too perfectly, with zero errors. The model captures every bit of noise in the data. This model has a high (perfect) variance and no bias.
Is this the model we want?
Probably not. With an increase in polynomial order the complexity of the model increases. There’s no guarantee that this model will perform just like that on other data.
So what a better model should look like? Something between too bad and too good? Let’s try lowering the polynomial degree from 6 to 2.
# polynomial fitpoly_fit = np.poly1d(np.polyfit(x,y, 2))plt.plot(x, poly_fit(x), linestyle='-')plt.scatter(x, y);
This 2-degree polynomial model has some errors and is certainly not as good as 6-degree, but it’s also not as bad as a simple linear fit. So it turns out that a higher degree polynomial fits the data better, but it also makes the model complex.
A complex model overfits training data but fails to make high-quality predictions in testing and out-of-sample data. A simple model, on the other hand, underfits training data and does not perform well in testing data either.
So data scientists aim to find a sweet spot to balance the tradeoff. But how exactly in practice?
Underfitting is easy to spot because it shows up in the model performance. Whereas, overfitting is difficult to catch because models show good performance in the error metrics. It is also really hard for a data scientist to throw away a really good fit model and choose a lower performing one. But there are some tools available to make this decision easier. Here’s a couple of options:
1) Visual interpretation of error vs complexity: Generally, a good practice is to choose a model is by plotting:
error vs complexity in training data
error vs complexity in testing data
examine where two curves intersect and tune parameters accordingly
In regression, the bias-variance tradeoff can be examined by plotting Mean Squared Error (MSE) against model complexity. In classification, the same can be examined by plotting the number of misclassification against complexity.
2) Regularization: It is a technique to optimize model performance by adding a small bias in the cost function. This small bias shrinks feature coefficients and reduces their sensitivity.
If there are too many features in a dataset, regularization controls their effects and makes them less sensitive. It is done by shrinking model coefficients towards zero. Two types of regularization are commonly used — L1 (LASSO regression) and L2 (Ridge regression) and they are controlled by a hyperparameter λ.
To summarize the concept of bias-variance tradeoff:
If a model is too simple and underfits the training data, it performs poorly in real prediction as well.
A model highly tuned on training data may not perform well either.
The bias-variance tradeoff allows for examining the balance to find a suitable model.
There are two ways to examine the tradeoff — a. visual interpretation of errors vs complexity and b. regularization.
Hope it was a useful discussion on this important concept in machine learning, if you have comments feel free to write them down below or connect with me via Medium, Twitter or LinkedIn.