Understanding Bias and Variance in machine learning through a philosopher’s perspective.
By: Siva Ram Krishna Nagi Reddy
Tags: Machine Learning, Philosophy, Andrew Ng, Bias, Variance

I am sure that you are confused with the title of this article. Just bare with me few minutes and all your confusion will be gone.

I am pretty sure you have came across with this type of image in many articles relating to bias and variance.

Bias and Variance visualization. Pic credits Andrew Ng

So lets first go through this image and understand what it is try to convey and then we will move to the philosopher part.

The axes which are labelled as X1 and X2 are the features and the cross and dot marks are the coordinate points which represents (X1,X2) at a given point.

The blue line that is plotted in the figure is the decision boundary that is a binary classifier( a classifier that classifies a given point as 1 or 0 , in our case dot or cross) which has learned from data in the training process.

Now comes the philosopher part.

First just imagine that the machine learning model or the binary classifier represents your perspective .

Now think a training sample represents a stranger around your house or office to whom you have never met.

So one day a local exhibition is being conducted where all the strangers gonna come. You also some how got an invitation.

On the day of the local exhibition you got know all these strangers through conversations or by observing . After the exhibition is over you returned to your home and started to think back who all from that group of strangers are good or not so good people depending on few aspects of them like behavior, occupation, talking style, dressing sense etc.(These can be called as the features of a machine learning model in our case features of a classifier).

So in real life if you combine all these features (behavior, occupation, talking style, dressing sense) that you relay on to tell if a person is good or not good. For instance consider a member from your family who might rely on same features to tell if a person is good or not good but might be putting more weight on behavior of a person to decide. Similarly you might be putting more weight on occupation to decide. In philosophy this is called as perspective or point of view which can differ from person to person.

Now comes the comparison part

In can of bias

If you learned or created a perspective after meeting the strangers at the local exhibition that if a person has a good dressing sense than he/she is a good person regardless of behavior, talking style, occupation (giving more weight to dressing sense than all other attributes).This rule is very simple to determine and have a great amount of bias towards the dressing sense attribute. So in general dressing sense is not very dependable attribute to decide if a person is good or not. Because incase if you are in a office every body will have a descent attrite and you will fail to decide when environment changes. And it might also not work on the people who attended the exhibition also. So you have to change your perspective.

This case is generally is called as Underfitting of a model in machine learning terms.

2 . In case of Variance

This case can be treated as exactly opposite of bias. Suppose you learned or created a perspective with some odd combination of attributes through which you are able to decide weather a person is good or not good with 99% accuracy who were all present in the local exhibition. It seems you have developed a rule particularly for the people who attended the exhibition. Which is a very nice perspective (very complex perspective), then what is the problem? Consider these questions. What if most of the people attend exhibition are generally good? What if most of the people around the exhibition are generally not good? So when you go to a different environment or place where the proportion of good and not so good people is 50:50 then you perspective gonna fail miserably.

This case is generally is called as Overfitting of a model in machine learning terms.

3 . Just Right

If you have understood the above two cases then it will be very easier to understand this one. Suppose you developed a perspective through which you able decide whether a person is good or not accurately, for people exhibition and people belonging to any other place or environment. Then I think your perspective is very reliable. Not leaning towards bias nor towards variance just in the middle of both of them. That’s why it is called Just Right.

This is generally called as an Ideal model in machine learning terms.

Comparisons :

Perspective to Machine learning model’s decision boundary .

to . People at local exhibition to Training Samples .

to . People belonging to different place or environment to Test Samples

to Behavior, Occupation, Talking style, Dressing sense to Features

I hope you now got a better sense of what is Bias and Variance through this article.

Please give your feedback in comments section.

Thank You.