Bias and Variance: the core concept of Machine Learning
By: Pooja Sharma
Tags: Machine Learning, Deep Learning, Bias Variance Tradeoff, Data Science

Bias and Variance are one of those concepts that are easily learned but difficult to master. In Machine Learning, when we want to optimize model prediction, it is very important to understand the parameters which describe errors and accuracy, In Machine Learning any model’s performance is based on its correct predictions and how well it is generalized on the training data(seen data), test data and validation data(unseen data) and real-time data.

Let’s see what this means:

Source: Google Images

Bias

In the first diagram, if we fit a straight line to the data, maybe we get a logistic regression fit to that. This is not a very good fit to the data and so this is a class of a high bias, what I say that this is underfitting the data.

In real life, suppose you have to give a test and you are provided with a cheat sheet but you are not interested in the learning so you just randomly answer questions correctly and got a 50% result. That is the problem of High Bias.

If the model is performing poorly over the test and the train test, then we call that an underfit model. An example of this situation would be building a linear regression model over non-linear data.

Variance

In the third diagram, If we fit an incredibly complex classifier may be a deep neural network or neural network with all the hidden units, maybe we can fit the data perfectly, but that doesn’t look like a great fit either. So there’s the classifier of high variance and this is Overfitting the data.

Now according to the previous example, there is a student who memorizes all the questions given in a cheat sheet but the test contained questions which he has never seen before. Therefore his performance went down significantly.

The situation where the model is performing well on the training data but the performance drops significantly over the test set is called an overfitting model.

There might be some classifier in between, with a medium level of complexity like (see diagram 2) that maybe fits it correctly.

So in a 2D example like this, with just two features, x1 and x2, we can plot the data and visualize bias and variance. In high dimensional problems, we can’t plot the data and visualize decision boundary, Instead, there are a couple of different metrics that help in understanding bias and variance.

To understand bias and variance two key numbers are train set error and dev set error. So continuing cat classification example where found a cat is a positive example and found dog is a negative example. We assume that humans get nearly 0% error or the optimal error or the Bayes error in predicting the cat.

Source: Google Images

So let’s say if the training set error is 1% and our dev set error is, 11%. S o we are doing well on the training example but we’re doing relatively poorly on the development set. So this looks like we might have overfitted the training set, that somehow we’re not generalizing well, to this whole cross-validation set which is a development set, this is high variance.

and our dev set error is, o we are doing well on the training example but we’re doing relatively poorly on the development set. So this looks like we might have the training set, that somehow we’re not generalizing well, to this whole cross-validation set which is a development set, this is If for example training set error is 15% and our dev set error is 16% then we know that model performs poorly on the training set then this is underfitting the data so the algorithm has a high bias . But in contrast, this actually generalizing at a reasonable level to the dev set whereas performance in the dev set is only 1% worse than the training set.

and our dev set error is then we know that model performs poorly on the training set then this is the data so the algorithm has a . But in contrast, this actually generalizing at a reasonable level to the dev set whereas performance in the dev set is only 1% worse than the training set. Now suppose, that we have a 15% training set error so that’s pretty high bias, but when you evaluate the dev set it does even worse, maybe it does 30% . So in this case algorithm has a high bias because it’s not doing that well on the training set, and has a high variance . So this has really the worst of both worlds.

training set error so that’s pretty high bias, but when you evaluate the dev set it does even worse, maybe it does . So in this case algorithm has a because it’s not doing that well on the training set, and has a . So this has really the worst of both worlds. And in the last example, if the training set error is 0.5% and the dev set error is 1% then we just have low bias and low variance.

The takeaway is that by looking at our training set error we can get a sense of how well we are fitting, at least the training data, and so that tells us if we have a bias problem. And then looking at how much higher our error goes, when we go from the training set to the dev set, that should give us a sense of how bad is the variance problem, so we’ll be doing a good job generalizing from a training set to the dev set, that gives us a sense of our variance. All this is under the assumption that the Bayes error is quite small and that our training and our dev sets are drawn from the same distribution

Recipe to improve the model’s performance

If the algorithm has a high bias, does not fit in the training set that well, some things we could try would be to try to pick a network, such as more hidden layers or more hidden units, or we could train it longer or try some more advanced optimization algorithms. we can try a lot of different neural network architectures and maybe we can find a new network architecture that’s better suited for this problem but there is no guarantee that this step will work but can be tried out. Whereas getting a bigger network almost always helps and training longer doesn’t always help, but it certainly never hurts.

Now when we look at dev set performance and the algorithm has high Variance, the best way to solve the problem of high variance is, to get more data but its’ not always easy to get more data. We could try regularization which is a very useful technique for reducing variance and we can also try a lot of different neural network architecture.