Bias and Variance in Machine Learning | by Arun Jagota | Towards Data Science

These concepts are important to both the theory and the practice of data science. They also come up in job interviews and academic exams.
A biased predictor is eccentric, i.e. its predictions are consistently off. No matter how well it’s trained, it just doesn’t get it. Generally, such a predictor is too simple for the problem at hand. It under-fits the data, no matter how rich.
A high-variance predictor is, in a sense, the opposite. It often arises when one tries to fix bias and over-compensates. One’s gone from a model that is too simple — i.e., biased — to one that is too complex — i.e. has high variance. It over-fits the data.
Somewhere between these two is the “sweet spot” — the best predictor. Often this is not easy to find. A data scientist can help. That’s another story ...
Example
The input is a temperature in (say) Fahrenheit. We want it classified comfortable or not. The training set may capture the judgments of a single person or a group.
As noted elsewhere, this problem is not linearly separable. Simply put, the correct solution should behave as
Too low     -> uncomfortable    (U)Just right  -> comfortable      (C)Too high    -> uncomfortable    (U)
It’s clear we need two thresholds, one to distinguish too low from just right, one to distinguish just rightfrom too high.
Bias
What if we are limited to use only one? Such as when our classifier is only capable of learning linear decision boundaries. No training set — no matter how rich — will help. We just can’t get the middle predicted comfortable and the two extremes predicted uncomfortable.
Towards A Better Solution
What if we had domain knowledge. That is, we knew the characteristics of the problem. Here that it needs two thresholds. We could work out a good solution.
We’ll suppose we don’t have such domain knowledge. Said another way, we want a generic solution.
Nearest Neighbor Approach
We’ll consider a single nearest neighbors approach because it can learn nonlinear decision boundaries and is simple to describe. We’ll describe it specialized to our problem. For two reasons: (i) further clarity and (ii) convenient for discussing its variance.
Predict the temperature to be comfortable if at least half the people labeled it comfortable; uncomfortable if not.
Now suppose the algorithm is trained on the following data set. Each temperature was labeled — C or U — by 10 people.
Temperature            10 ... 45 50 55 60 ... 80 ... 100 ..% who Labeled C         0 ... 50 30 40 70 ... 70 ... 0
This algorithm would label 45 as C, 50 and 55 as U, and 60 as C. This doesn’t seem right. 50 and 55 are deemed uncomfortable. We expect 45 should also be.
What is going on here? We have only 10 labelers. It’s plausible that “half of them labeling 45 as C” is noise. With many more labelings, this fraction might well drop to below 0.5.
Another way to look at this is that the model’s predictions have high variance. Think this way. Imagine a different set of 10 people labeling the data. Train on this new set. The prediction at 45 is unlikely to be off. This would be too much of a coincidence. That said, there very well could be some temperature on which the prediction is off.
So, as we’ve seen in our example, a high variance predictor is susceptible to learning the noise in the training set. Noise doesn’t repeat itself. Next time we train it, it will learn noise on a different input. From the point of view of a user, who only uses it as a black-box predictor, the predictions aren’t consistent across training runs.
Reducing Variance
So how do we fix this algorithmically . That is, without asking for additional labeling (which incurs a cost). We smooth the data, either during pre-processing, or as part of the algorithm. Below we opt for the latter. We replace single nearest neighbor by three. For simplicity, ignoring computational efficiency considerations, we describe it in the form below:
For the input temperature T, get all the labelings of T, T-1, and T+1. Predict T’s label to be the majority label among all these.
We have a more robust classifier. Each prediction will be based on a labeled sample of size 30 — three times more than we had from our single nearest neighbor classifier. Compared to the single threshold classifier, this one is less biased. It is inherently still capable of learning non-linear decision boundaries.
Of course, precisely how accurate this new classifier is compared to the previous two is an empirical question. This involves a precise comparison of the combined effects of bias and variance of each classifier. In short, it needs a suitable train-test empirical evaluation. That’s another story.