What Bias-Variance Bulls-Eye Diagram Really Represents | by Angela Shi | Towards Data Science

When looking for Bias-Variance tradeoff, we can see the following bulls-eye diagram:
They seem intuitive and eye-catching to represent the bias and the variance, with some blue dots. But what do they really mean? The central point represents the target, but what do the blue dots really represent? And how these bulls-eyes are related to underfitting and overfitting?
The bias-variance tradeoff is related to the notions of overfitting and underfitting. To illustrate, we can take the example of Polynomial Regression. In the following example, the real relationship between x and y is quadratic
So if we use a simple linear regression, the model is underfitted;
If the degree of the polynomial is too high (8 for example), then the model is overfitted.
And the optimal model is of degree 2.
Now if we go back to the bulls-eye diagram, the relationship is stated as follows:
The optimal model corresponds to low variance and low bias. (OK, understandable)
Overfitting corresponds to high variance and low bias. (But why? how is it related to 8th-degree polynomial regression as seen in the previous digram?)
Underfitting corresponds to high bias and low variance. (But why? how is it related to the simple regression as seen in the previous diagram?)
And what about high variance and high bias? What model would lead to this situation?
If we put them together, how can we relate the dots in the bulls-eye to the data points in the graphics of the regressions?
At the first glance, the bulls-eye diagram is quite intuitive and eye-catching. But if we really try to ask this question to ourselves: each prediction point has two coordinates? What are the two dimensions?
Then we realize that the eyes are just an artistic representation of the target variable values. Because we usually have only one target variable. So if we want to be more accurate, we should only show one dimension.
The red dot on the middle is the real value (of one specific observation) we try to predict. And the blue dots are the predictions of different models we build.
So if we want to describe the comparison between the target value and the predictions more accurately, the bulls-eye would become only one axis, which represents the target variable.
Now, what do they really represent, how can we create a concrete example of model? Let’s code.
We are going to create multiple models to illustrate the relationship between (underfitting and overfitting) and (bias-variance tradeoff), in the following process:
First, we choose to generate some simple data from a known function.
Then we can try to create models
For one or multiple new observations, we can do the prediction and see the variance and bias.
We can generate some data that has a true relationship that is quadratic.
import numpy as npimport pandas as pdn = 20x = np.linspace(0,7,n) + np.random.randn(n)*0.1def f(x):    return (x-3)**2y = f(x)+np.random.randn(n)
And we can plot the data, with the real function that is red.
To create underfitted models with the previous data, we can create simple linear regression. With some training data chosen from the original dataset, we can create some models.
for i in range(20):    model = make_pipeline(PolynomialFeatures(1), LinearRegression())    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)    model.fit(X_train, y_train)    y_plot = model.predict(X_seq)    plt.plot(X_seq, y_plot,c=”c”)    plt.scatter(4, model.predict([[4]]),s=300,c=”c”)
If we do predictions for the value of 4, we can get the following plot:
the red dot is the real target value
the blue dots are the predictions of the models (simple linear regressions) created by samples of training data.
Now, we can clearly see the bias. And we can do the predictions for several values
And now, let’s do a recap for different illustrations:
For overfitting, we can create the same comparison, by changing the degree of the polynomial.
for i in range(20):    model = make_pipeline(PolynomialFeatures(8), LinearRegression())    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)    model.fit(X_train, y_train)    y_plot = model.predict(X_seq)    plt.plot(X_seq, y_plot,c=”c”)    plt.scatter(4, model.predict([[4]]),s=300,c=”c”)
When the degree of the polynomial is 2, we can can clearly that the predictions of the models have low biais and low variance.
We can also illustrate with a regression tree
I hope that now you see the bias-variance bulls-eye in a more accurate way.
If so, one last thing: what is the model that could have high variance and high biais?