Understanding the Bias — Variance Trade-off: With Examples and a Simple Explanation
By: Meet Raval
Tags: Machine Learning, Bias, Bias Variance Tradeoff, Variance, Data Science

Bias & variance are some of those topics which are easy to understand but challenging to master. So, in this tutorial, I will explain bias & variance such that you don’t have to worry about it ever again. Without wasting much time, let’s get straight to the point.

Bias 101:

Bias is related to the training set error and it is also related to Under-fitting.

Let’s understand what does I mean from the above statement regarding bias.

Imagine you have a classifier, any classifier, and you trained the classifier on the train set and then calculated the train set error; then, if the train set error is high, we can say that our model has high bias. And if the train set error is low, then we can say that our model has low bias.

In a nutshell, the high train set error corresponds to High Bias. And the main reason behind the High Train Set error is that our model is under-fitting the Train data. That’s why High Bias is also related to the Under-fitting.

Variance 101:

Variance is related to the Test set error (How big is the Test set error compared to Train set error) and it is also related to the over-fitting.

Let’s again understand what I mean by the above statement regarding variance.

After training the classifier and obtaining the Train Set Error, we have to get the Test set error and compare both Train & Test Set errors with each other.

If the difference between Train & Test set error is more (Test set error is much higher compared to the Train set error) then, we can say that we have High Variance. Obtaining a training set error of 0.5% and a Test set error of 10% means our model is not generalizing well, which means that our model is Over-fitting as well.

In a nutshell, we have high Variance, when the Test set error is much bigger compared to the Train set error. And when that happens, our model is over-fitting as well.

Bias-Variance Trade-Off 101:

Bias-Variance trade-off comes into the picture because,

Decreasing Bias will increase Variance & Decreasing Variance will increase Bias.

In a nutshell, if you will try to decrease Bias too much, then that will result in the model being more Complex and Less Generalized. Meaning you will Over-fit the model.

And if you will try to decrease the Variance too much, then that will result in a more generalized and less complicated model. Meaning you will Under-fit the model.

So, our goal is to find the right balance between Bias & Variance.

Visualizing the Different Bias-Variance scenarios:

Now, for better understanding, let’s visualize different Bias-Variance scenarios on a data-set containing two dimensions (x1 & x2) and two labels (o and x). We discuss different decision boundaries for all the different Bias-Variance situation.

This Figure has four different Bias-Variance situations that you can find your self in. Let’s visualize them for better understanding.

Low Bias & High Variance:

If we have low Train set error(0.5%) and Test set error(10%) is high. Then we have Low Bias and High Variance condition, which is also known as Over-fitting of the model. And the example of the Low Bias & High Variance can be seen in the two dimensions, two label data-set.

High Bias & Low Variance:

If we have a High Training set error(13%) and Test set error (13.50%) is not that high compared to the Training set error, then we have High Bias and Low Variance situation. Also known as Under-fitting. The example of which on the two dimensions & 2 label data set can be seen above.

High Bias & High Variance: The worst case scenario.

If we have High Training Set error(12%) and High Test set error(20%), then we have High Bias and High Variance condition. Which is a combination of Under and Over-fitting? The example of which can be seen in the above image with two dimensions & 2 label data set shown above.

Low Bias & Low Variance: This is what we want.

If you have Low Training set error(0.5%) and Low Test set error (1%), then we can say we have the perfect model with Low Bias & Low Variance. The example of which can be seen above.

Solving High Bias & High Variance Problem:

If you have a High Bias or High Variance problem, then trying the above solutions will work. But beware! When you are trying to reduce Bias, then keep an eye on Variance because you don’t want to make Bias Extremely low because that will increase Variance. It’s all about finding that sweet spot for Bias and Variance.

Conclusion:

So, you will have to deal with Bias-Variance Trade-off and now that you understand what Bias- Variance trade-off is and how to solve the Bias- Variance problem. I think you will be more than comfortable dealing with this stuff.

I hope you don’t have any doubts now. If you have any doubts then feel free to comment below, will reply within 24 hours.

And If you don’t have any doubts and loved reading it or it helped you in any way, then please make it clap. It motivates me in this depressing Covid-19 time.

Be well and safe :)