Why is my model performing poorly?
By: Martin Šiklar
Tags: Python, Data Science, Regression, Machine Learning

Why is my model performing poorly?

Photo by Dana Vollenweider on Unsplash

Are you sometimes struggling with your model performance? Is your model performing well during training but when you apply it to new data, it fails? Or do you even struggle finding a good model in the first place because it just doesn’t seem to learn enough from the data you provided?

If any of the statements above apply to your situation then your model might be either overfitting or underfitting the training data. If you continue reading, you will learn the necessary knowledge on how to identify these phenomenons and be provided with tools and remedies on how to overcome them and finally increase your models performance — the true goal of every machine learning engineer. We will be looking into this topic from the Regression perspective with code examples provided in Python and scikit-learn. Please note that this article is highly inspired by the excellent book “Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition” which is a great ressource for boosting your machine learning knowledge.

Basics

As an introduction we will start by briefly looking into examples of models which are underfitting, overfitting and performing well. To do this, let’s start with the simplest possible regression model to visually illustrate what over-and underfitting means: linear regression.

Underfitting

Imagine we have some non-linear data based on a quadratic function:

import numpy as np n = 100

X = 4 * np.random.rand(n, 1) - 2 #Quadratic Function: y = ax^2 + bx + c

y = 0.6 * X**2 + X + 3 + np.random.rand(n, 1) #adding random noise

It is clear that a simple linear model will never fit this quadratic data properly, but let’s do the exercise for demonstration purposes:

from sklearn.linear_model import LinearRegression

linear_reg = LinearRegression()

linear_reg.fit(X,y)

Linear model underfitting quadratic data — Image by Author

You can see that the linear function was able to detect the general trend of the data but the non-linearity is (as expected) not captured at all by the model. The model is clearly underfitting the data and doesn’t depict the underlying relationship. The models performance is low.

The “perfect” fit

Let us therefore try out a polynomial regression which is able to capture more complex relationships in data. The easiest way of implementing a polynomial regression is to simply add powers (in our case square because we used a quadratic function) of each feature as a new feature and then apply the same Linear Regression function we used above.

from sklearn.preprocessing import PolynomialFeatures #add power of two to the data

polynomial_features = PolynomialFeatures(degree = 2, include_bias = False)

X_polynomial = polynomial_features.fit_transform(X) #print initial feature

print(X[0]) [-1.52521333] #print feature after applying polynomial transformation

print(X_polynomial[0]) [-1.52521333 2.32627571]

We can see that X_polynomial now contains the initial feature (-1.52521333) as well as its square (2.32627571). If we feed this in our Linear Regression function we end up having a Polynomial Regression. Let’s see how it fits to the data.

polynomial_reg.fit(X_polynomial, y)

Polynomial model fitting quadratic data — Image by Author

We can inspect the estimated coefficients of the model and compare them to the coefficients we used during the definition of our quadratic function.

polynomial_reg.intercept_ array([3.51521119]) polynomial_reg.coef_ array([[1.01029724, 0.61907757]])

The models performance is very good! The polynomial regression estimated coefficients that are very close to the real coefficients, only the intercept was not well captured by the model due to the noise we added in the beginning.

Estimated model coefficients

Real coefficients of quadratic function

The general assumption is that a polynomial regression will fit the data much better than a simple linear regression. Let’s look at another example and use a higher degree polynomial model to fit our data.

Overfitting

As can be seen in the image below, a higher degree (100) polynomial model is highly overfitting the data. You can clearly see how the model wiggles around at the edges trying to fit the training data as good as possible.

Higher degree polynomial model overfitting quadratic data — Image by Author

It is clear that from our three examples (linear, polynomial degree 2 and polynomial degree 100) our 2nd degree polynomial model performed the best because the underlying data was created using a quadratic function.

However, normally we don’t know what function represents our data in the best way possible or what model might be the best fit for us. So how do we find the best fitting model and what signs can we follow to judge a models performance?

Finding the clues

Photo by Austin Chan on Unsplash

Option 1: Diverging Cross validation and training errors

You might be familiar with this option since it is one of the most frequent you will find in literature: Cross Validation. Here the general idea is to split your training data set into k smaller subsets (folds). Subsequently you train your model on k-1 of these subsets and evaluate the models performance on the remaining subset. You repeat this procedure k times and end up with k evaluation scores. To get the overall cross-validation score you can simply take the mean value of the k scores.

In scikit-learn this method is called K-fold cross validation and the underlying idea is to check how the model performs on the training set as compared to the cross validation scores. If a model performs well on the training set but the cross validation scores are low the model is overfitting. If the model performs poor on the training set and equally poor according to the cross validation scores, it is underfitting. The capability to correctly predict new data the model has never seen before is often referred to as generalization in literature. Let us go through some examples!

Root mean square error of the simple linear regression

Firstly, let’ us compute the root mean square error (RMSE) of our linear regression model predictions on the training data.

from sklearn.metrics import mean_squared_error

np.sqrt(mean_squared_error(y, linear_reg.predict(X))) 0.7736957615287954

This means that our models prediction on average diverges from the real data by about 0.77 which is our training error. Let’s compute the cross validation scores of our linear model. In this example we will be using the cross_val_score implementation in scikit-learn with 10 folds.

from sklearn.model_selection import cross_val_score lin_reg_scores = cross_val_score(linear_reg, X, y, scoring = “neg_mean_squared_error”, cv = 10)

What we get as an output are ten output scores measuring our models scores during cross validation. Each of these models was trained an a different subset of 90% of the data and evaluated with the remaining 10 % of the “unseen” data. In total this was done 10 times:

array([-0.71517651, -0.83664222, -0.631302 , -0.550304 , -0.55877713,

-0.56387244, -0.73492102, -0.55389454, -0.49322261, -0.67008695])

To get the aggregated models cross validation error which we can compare to the root mean squared error during the training, we can simply take the square root and subsequently the mean of the scores:

np.mean(np.sqrt(-lin_reg_scores)) 0.7917563622242139

Finally we get to a mean cross-validation RMSE of 0.79. If we compare this error to the training set error we can see a difference of about 0.02. It is normal that the cross validation error is higher than the train error, but is 0.02 good or bad? Is it high or low? How can we tell? Hang on for another example and you will know!

Root mean square error of the 2nd degree polynomial regression

Let us do the same exercise for our 2nd degree polynomial regression and see how these errors compare to the errors of the linear model.

np.sqrt(mean_squared_error(y,polynomial_reg.predict(X_polynomial))) 0.2741445060567453

Wow! 0.27 is much lower than the 0.77 RMSE we calculated from the linear model. We can clearly see that the linear model was clearly underfitting the data because its training error is much higher than the training error of the polynomial regression. Let us now compute the cross validation error of the polynomial regression:

poly_reg_scores = cross_val_score(polynomial_reg, X_polynomial, y, scoring = “neg_mean_squared_error”, cv = 10)

np.mean(np.sqrt(-poly_reg_scores)) 0.28120221324092176

For the polynomial model we get a cross validation RMSE of 0.28 which is only off by 0.01 to the training data RMSE. We can therefore conclude:

The linear regression RMSE is much higher than the RMSE of the polynomial regression in our example. Furthermore, we can see that the difference in RMSE between the training data and the cross validation is higher for the linear regression (0.02 > 0.01). This two signs yield the conclusion that the the linear regression is underfitting the data.

Root mean square error of the higher degree polynomial regression

Last but not least, let us look at the error of the higher degree (100) polynomial regression which gets an RMSE of stunning 0.24 on the training data which is even better than the second degree polynomial RMSE above!!

np.sqrt(mean_squared_error(y,higher_polynomial_regression.predict(X_higher_polynomial))) 0.24841856155222344

But how does the cross-validation RMSE look like? Yep, its a jaw-dropping RMSE of 1020471.12. The model is heavily overfitting the training data (it has the lowest RMSE of all models) but performs horribly on unseen data as indicated by the unbelievably high cross validation RMSE. This is a text book example for strong overfitting. In machine learning terms the model therefore has a poor ability to generalize.

higher_poly_reg_scores = cross_val_score(higher_polynomial_regression, X_higher_polynomial, y, scoring = “neg_mean_squared_error”, cv = 10)

np.mean(np.sqrt(-higher_poly_reg_scores)) 1020471.1265272169

Interpreting the signs

Below you can find a summary on how you can interpret the errors and draw conclusions if your model is potentially over- or underfitting the training data:

High Training Error & Even Higher Cross Validation Error → Sign for underfitting

Low Training Error & Slightly Higher Cross Validation Error → Sign for a good fit

Very low training Error & Very high Cross Validation Error → Sign for overfitting

Option 2: Learning Curves

Another possibility, on how to detect over- and underfitting models is to look at their learning curves. Learning curves visualise the relationship of the models performance in regard to training data size. Let’s look at two learning curves to explain the basic principle behind this analysis. For this part we will be using the same data from above ,which were generated using a quadratic function.

Underfitting Learning Curves

Let’s first divide our training data into a smaller training and validation set.

#split training set into training and validation set

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=4)

train_errors = [] #append training errors for plot

val_errors = [] #append validation errors for plot

Now let’s loop through the observations in the new training set, starting with only one observation. Based on this, we will train our linear regression model and evaluate its performance using the entire validation set. We will do this iteratively and add another observation to the training set each round, followed by a retraining of the model and another evaluation with the validation set:

#loop over each observation and iteratively compute RMSE for the new training and the entire validation set

#in each iteration add one more observation to the training set

for i in range(1, len(X_train)):

linear_reg.fit(X_train[:i], y_train[:i])

y_train_predicted = linear_reg.predict(X_train[:i])

y_val_predicted = linear_reg.predict(X_val)

train_errors.append(mean_squared_error(y_train[:i], y_train_predicted))

val_errors.append(mean_squared_error(y_val, y_val_predicted)) plt.plot(np.sqrt(train_errors), “r-”, linewidth=2, label=”trainining set”)

plt.plot(np.sqrt(val_errors), “b-”, linewidth=2, label=”validation set”)

plt.legend(loc=”upper right”, fontsize=15)

plt.xlabel(“Training / Validation set size”, fontsize=15)

plt.ylabel(“Root Mean Square Error”, fontsize=15)

Let us have a closer look at the produced graph and start with the RMSE on the training set (red line). Initially with only a few instances the training error is very low because the linear model can fit one or two instances very well, but once the training set starts to get bigger, the RMSE starts to rise very quickly until it reaches a constant level at about 30 instances. At this point the algorithm stopped learning what it could (remember that our data is quadratic and our model is linear) and increasing the training data size doesn’t help anymore. The validation set error (blue line) is initially very high because the linear model is only trained on a handful of instances. However it rapidly gets smaller and smaller with every new instance that is added to the training set until it quickly reaches a plateau at a slightly higher level than the training set error. These types of curves are characteristic for models that are underfitting the data.

Learning Curves for an underfitting model — Image by Author

Overfitting Learning Curves

Let us do the same exercise but by replacing the linear model with a higher degree polyonimal regression. When we look at the new curves it is hard to really tell a difference but we have to look for the subtle sings that tell us that our model is overfitting: The training error of this model is generally much lower than the training error of the previous linear model. Furthermore, for a very long time there is a larger gap between the red and blue line— as already discussed above, this is one of the main signs that your model is overfitting: A high validation error and a low training error.

A very interesting detail however can be seen on the very end of the blue line: With more training instances the validation error starts dropping significantly and ends up being quite close to the training error — resulting in a much better model in the end.

Learning Curves for an overfitting model — Image by Author

What can I do when my model is over- or underfitting?

This might be maybe the most important questions and luckily I can leave you with a set of strategies that will hopefully help you building better performing models.

Overfitting:

Use more data! As we could see on the last learning curve example, if you keep adding new features to your training data set, the validation error will eventually go down and the model will stop overfitting. Ofter however your training data set is limited so you will need a different strategy.

As we could see on the last learning curve example, if you keep adding new features to your training data set, the validation error will eventually go down and the model will stop overfitting. Ofter however your training data set is limited so you will need a different strategy. Regularization: In other words, put some constraints on your model and lower the degrees of freedom.The more constraints you put on your model (the fewer degrees of freedom your model has) the harder it will be to overfit the data. In our example this means lowering the polynomial degree for example. In terms of regression, you should also look into Ridge, Lasso or Elastic Net Regression which put different constraints on their model weights.

Underfitting:

Use more data! Yep, also in this case more data will very likely boost your models performance to a certain extent. However, if your assumptions about the data are wrong and you decide to use a classifier that is not able to capture the real nature of the data, then simply adding more data wont cut it!

Yep, also in this case more data will very likely boost your models performance to a certain extent. However, if your assumptions about the data are wrong and you decide to use a classifier that is not able to capture the real nature of the data, then simply adding more data wont cut it! Increase the complexity: In the case of underfitting, putting less constraints on your model and increasing its complexity will very likely result in a better model. Increasing complexity could mean going beyond a simple linear model and making use of polynomial models, or maybe even tree based models.

Of course there is more strategies to overcome this, but data amount and model complexity can be seen as the smallest common denominator of the most of them. Before you however start switching models and changing their complexity, you should first check if your model is over- or underfitting so you can start heading into the right direction. By now you should have the necessary tools to do so in your backpack!

Photo by Holly Mandarich on Unsplash

If you want to learn more on this, please visit the links below where I have place some excellent literature and articles on this and related topics.

Further material: