Understanding bias and variance. Anyone learning a data science 101... | by Vinodh Kumar Ravindranath | Towards Data Science

Anyone learning a data science 101 course is confronted with these terms bias and variance that define the accuracy of the machine learning model. I found some of the material around it confusing, so I thought I will try to explain with some illustration and code.
Before I get to the code, basically bias is nothing but the expected error (average of the error) in the model when applied on different test data sets. Let me use the dartboard analogy. The numbers in the circles represent the error till the circumference of the circle where it is written.
In board A, the average error or bias is equal to (0 + 0 + 0 + 1 + 2) / 5 = 0.6
Even though Board B looks different from Board A, it has the same average error or bias = (0 + 0 + 1 + 1 + 1) / 5= 0.6
Now the question is which dart thrower would you bet on? one using board A or board B. This is where variance comes into the picture.
Variance is the variance of the error when applied on different test data points. Lets now try to apply this concept on the two dart throwers using Board A and B respectively.
Board A’s variance = (0 + 0 + 0 + 12 + 22) / 5 = 1.0
Board B’s variance = (0 + 0 + 12 + 12 + 12) / 5 = 0.6
This indicates that while board A dart thrower hits the bull’s eye 50% more number of times than the board B dart thrower, he is much more erratic when he does not get it right. On the other hand, board B dart thrower has a steady hand and always hovering on or around the bull’s eye mark. In the world of machine learning models, usually models that are similar to the board B dart thrower are preferred since it avoid embarrassing mistakes and is more steady.
Turns out the the behaviour similar to Board B occurs in overfitted machine learning models (overfitted with respect to a training set) — i.e. variance will be high even if the bias of the model is low. So greater the variance you will tend to be worried pushing it to production since when it goes wrong, it can go wrong pretty badly.
In a model that is not overfitted (probably regularized), variance will be low — this means that the errors encountered on the test sets are not going to be ridiculous.
I have illustrated this by applying both linear and cubic regression to a linear data set with random perturbations.
Going through the code below (hopefully self explanatory) will help you see that though the average error (bias) is somewhat similar for both cubic and linear regression (similar to our dart throwers), however the variance is too high for cubic regression that makes it resemble the brilliant but erratic dart thrower on Board A.
import numpy as np
import matplotlib.pyplot as plt

# Generate points that are around a line. Adding noise 
final_point = 7500
num_points = 300
middle = 4
last = 15

x = np.random.uniform(0, final_point, num_points)
# y = 3x + 90
y = 3 * x + 90

p = 0.3

# Add random perturbations
# Big perturbations in 30% of the points
# Small perturbations at every point.
y += (np.random.binomial(1, p, size=num_points) * \
     np.random.normal(loc=0.0, scale=10000, size=num_points))\
     + \
     (np.random.normal(loc=0.0, scale=final_point / 5, size=num_points))
        
    

# Plot of Training Data 
plt.scatter(x, y) 
plt.xlabel('x') 
plt.xlabel('y') 
plt.title("Training Data") 
plt.show()

from sklearn.preprocessing import PolynomialFeatures
from sklearn import linear_model

# Curve fitting function
# Fits both cubic and linear curves and returns the prediction for both in a tuple
def fit_curves(sampled_x, sampled_y):  
    lin = linear_model.LinearRegression()
    lin.fit(sampled_x, sampled_y)
    linear_ypred = lin.predict(x[:, np.newaxis])
      
    poly = PolynomialFeatures(degree = 3) 
    sampled_x_poly = poly.fit_transform(sampled_x)
    x_poly = poly.fit_transform(x[:, np.newaxis])
    lin2 = linear_model.LinearRegression() 
    lin2.fit(sampled_x_poly, sampled_y)
    cubic_ypred = lin2.predict(x_poly)
    return((linear_ypred, cubic_ypred))

# Generate 1000 training sets
num_trials = 5
all_cpred = []
all_cbiases = []
all_lpred = []
all_lbiases = []

# Two figures - one each for cubic and linear regression
for j in range(2):
    plt.figure(j)
    plt.scatter(x, y, color = 'blue')


for i in range(num_trials):
    # Create a training data set
    sampled_indices = np.random.choice(num_points, size=25)
    sampled_x = x[sampled_indices]
    sampled_y = y[sampled_indices]
    sampled_x = sampled_x[:, np.newaxis]
    sampled_y = sampled_y[:, np.newaxis]
    (lpred, cpred) = fit_curves(sampled_x, sampled_y)
    
    # Plotting the sampled points - training sample
    for j in range(3):
        plt.figure(j)
        plt.scatter(sampled_x, sampled_y, color = 'green')
    
    # Plot results of cubic curve
    plt.figure(0)
    plt.scatter(x, cpred, color = 'red')
    
    # Plot results of linear curve
    plt.figure(1)
    plt.scatter(x, lpred, color = 'yellow')
    
    cubic_bias = np.abs(y[:, np.newaxis] - cpred)
    linear_bias = np.abs(y[:, np.newaxis] - lpred)
    all_cpred.append(cpred)
    all_cbiases.append(cubic_bias)
    
    all_lpred.append(lpred)
    all_lbiases.append(linear_bias)
    
plt.show()
print("Variance (cubic regression) = ", np.average(np.var(all_cpred, axis=0)))
print("Variance (linear regression) = ", np.average(np.var(all_lpred, axis=0)))
print("Bias (cubic regression) = ", np.average(all_cbiases))
print("Bias (linear regression) = ", np.average(all_lbiases))
#print(all_cbiases)

Variance (cubic regression) =  5896095.929190446
Variance (linear regression) =  1256507.0010197319
Bias (cubic regression) =  4223.96882929893
Bias (linear regression) =  3614.2993395834055

In this case, this would mean that the data scientist should choose the linear regression model for roll out to production.
In conclusion, as data scientists, it is important not to look at the average error (bias), but also pay attention to the variance. Higher the variance, it means that while there are cases where the model is super-accurate, there are also cases where the model goes horribly wrong, which you would like to avoid.