Why is my model performing poorly? | by Martin Šiklar | Towards Data Science

Are you sometimes struggling with your model performance? Is your model performing well during training but when you apply it to new data, it fails? Or do you even struggle finding a good model in the first place because it just doesn’t seem to learn enough from the data you provided?
If any of the statements above apply to your situation then your model might be either overfitting or underfitting the training data. If you continue reading, you will learn the necessary knowledge on how to identify these phenomenons and be provided with tools and remedies on how to overcome them and finally increase your models performance — the true goal of every machine learning engineer. We will be looking into this topic from the Regression perspective with code examples provided in Python and scikit-learn. Please note that this article is highly inspired by the excellent book “Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition” which is a great ressource for boosting your machine learning knowledge.
As an introduction we will start by briefly looking into examples of models which are underfitting, overfitting and performing well. To do this, let’s start with the simplest possible regression model to visually illustrate what over-and underfitting means: linear regression.
Imagine we have some non-linear data based on a quadratic function:
import numpy as npn = 100X = 4 * np.random.rand(n, 1) - 2#Quadratic Function: y = ax^2 + bx + cy = 0.6 * X**2 + X + 3 + np.random.rand(n, 1) #adding random noise
It is clear that a simple linear model will never fit this quadratic data properly, but let’s do the exercise for demonstration purposes:
from sklearn.linear_model import LinearRegressionlinear_reg = LinearRegression()linear_reg.fit(X,y)
You can see that the linear function was able to detect the general trend of the data but the non-linearity is (as expected) not captured at all by the model. The model is clearly underfitting the data and doesn’t depict the underlying relationship. The models performance is low.
Let us therefore try out a polynomial regression which is able to capture more complex relationships in data. The easiest way of implementing a polynomial regression is to simply add powers (in our case square because we used a quadratic function) of each feature as a new feature and then apply the same Linear Regression function we used above.
from sklearn.preprocessing import PolynomialFeatures#add power of two to the datapolynomial_features = PolynomialFeatures(degree = 2, include_bias = False)X_polynomial = polynomial_features.fit_transform(X)#print initial featureprint(X[0])[-1.52521333]#print feature after applying polynomial transformationprint(X_polynomial[0])[-1.52521333  2.32627571]
We can see that X_polynomial now contains the initial feature (-1.52521333) as well as its square (2.32627571). If we feed this in our Linear Regression function we end up having a Polynomial Regression. Let’s see how it fits to the data.
polynomial_reg.fit(X_polynomial, y)
We can inspect the estimated coefficients of the model and compare them to the coefficients we used during the definition of our quadratic function.
polynomial_reg.intercept_array([3.51521119])polynomial_reg.coef_array([[1.01029724, 0.61907757]])
The models performance is very good! The polynomial regression estimated coefficients that are very close to the real coefficients, only the intercept was not well captured by the model due to the noise we added in the beginning.
The general assumption is that a polynomial regression will fit the data much better than a simple linear regression. Let’s look at another example and use a higher degree polynomial model to fit our data.
As can be seen in the image below, a higher degree (100) polynomial model is highly overfitting the data. You can clearly see how the model wiggles around at the edges trying to fit the training data as good as possible.
It is clear that from our three examples (linear, polynomial degree 2 and polynomial degree 100) our 2nd degree polynomial model performed the best because the underlying data was created using a quadratic function.
However, normally we don’t know what function represents our data in the best way possible or what model might be the best fit for us. So how do we find the best fitting model and what signs can we follow to judge a models performance?
You might be familiar with this option since it is one of the most frequent you will find in literature: Cross Validation. Here the general idea is to split your training data set into k smaller subsets (folds). Subsequently you train your model on k-1 of these subsets and evaluate the models performance on the remaining subset. You repeat this procedure k times and end up with k evaluation scores. To get the overall cross-validation score you can simply take the mean value of the k scores.
In scikit-learn this method is called K-fold cross validation and the underlying idea is to check how the model performs on the training set as compared to the cross validation scores. If a model performs well on the training set but the cross validation scores are low the model is overfitting. If the model performs poor on the training set and equally poor according to the cross validation scores, it is underfitting. The capability to correctly predict new data the model has never seen before is often referred to as generalization in literature. Let us go through some examples!
Root mean square error of the simple linear regression
Firstly, let’ us compute the root mean square error (RMSE) of our linear regression model predictions on the training data.
from sklearn.metrics import mean_squared_errornp.sqrt(mean_squared_error(y, linear_reg.predict(X)))0.7736957615287954
This means that our models prediction on average diverges from the real data by about 0.77 which is our training error. Let’s compute the cross validation scores of our linear model. In this example we will be using the cross_val_score implementation in scikit-learn with 10 folds.
from sklearn.model_selection import cross_val_scorelin_reg_scores = cross_val_score(linear_reg, X, y, scoring = “neg_mean_squared_error”, cv = 10)
What we get as an output are ten output scores measuring our models scores during cross validation. Each of these models was trained an a different subset of 90% of the data and evaluated with the remaining 10 % of the “unseen” data. In total this was done 10 times:
array([-0.71517651, -0.83664222, -0.631302  , -0.550304  , -0.55877713,       -0.56387244, -0.73492102, -0.55389454, -0.49322261, -0.67008695])
To get the aggregated models cross validation error which we can compare to the root mean squared error during the training, we can simply take the square root and subsequently the mean of the scores:
np.mean(np.sqrt(-lin_reg_scores))0.7917563622242139
Finally we get to a mean cross-validation RMSE of 0.79. If we compare this error to the training set error we can see a difference of about 0.02. It is normal that the cross validation error is higher than the train error, but is 0.02 good or bad? Is it high or low? How can we tell? Hang on for another example and you will know!
Root mean square error of the 2nd degree polynomial regression
Let us do the same exercise for our 2nd degree polynomial regression and see how these errors compare to the errors of the linear model.
np.sqrt(mean_squared_error(y,polynomial_reg.predict(X_polynomial)))0.2741445060567453
Wow! 0.27 is much lower than the 0.77 RMSE we calculated from the linear model. We can clearly see that the linear model was clearly underfitting the data because its training error is much higher than the training error of the polynomial regression. Let us now compute the cross validation error of the polynomial regression:
poly_reg_scores = cross_val_score(polynomial_reg, X_polynomial, y, scoring = “neg_mean_squared_error”, cv = 10)np.mean(np.sqrt(-poly_reg_scores))0.28120221324092176
For the polynomial model we get a cross validation RMSE of 0.28 which is only off by 0.01 to the training data RMSE. We can therefore conclude:
The linear regression RMSE is much higher than the RMSE of the polynomial regression in our example. Furthermore, we can see that the difference in RMSE between the training data and the cross validation is higher for the linear regression (0.02 > 0.01). This two signs yield the conclusion that the the linear regression is underfitting the data.
Root mean square error of the higher degree polynomial regression
Last but not least, let us look at the error of the higher degree (100) polynomial regression which gets an RMSE of stunning 0.24 on the training data which is even better than the second degree polynomial RMSE above!!
np.sqrt(mean_squared_error(y,higher_polynomial_regression.predict(X_higher_polynomial)))0.24841856155222344
But how does the cross-validation RMSE look like? Yep, its a jaw-dropping RMSE of 1020471.12. The model is heavily overfitting the training data (it has the lowest RMSE of all models) but performs horribly on unseen data as indicated by the unbelievably high cross validation RMSE. This is a text book example for strong overfitting. In machine learning terms the model therefore has a poor ability to generalize.
higher_poly_reg_scores = cross_val_score(higher_polynomial_regression, X_higher_polynomial, y, scoring = “neg_mean_squared_error”, cv = 10)np.mean(np.sqrt(-higher_poly_reg_scores))1020471.1265272169
Below you can find a summary on how you can interpret the errors and draw conclusions if your model is potentially over- or underfitting the training data:
High Training Error & Even Higher Cross Validation Error → Sign for underfitting
Low Training Error & Slightly Higher Cross Validation Error → Sign for a good fit
Very low training Error & Very high Cross Validation Error → Sign for overfitting
Another possibility, on how to detect over- and underfitting models is to look at their learning curves. Learning curves visualise the relationship of the models performance in regard to training data size. Let’s look at two learning curves to explain the basic principle behind this analysis. For this part we will be using the same data from above ,which were generated using a quadratic function.
Let’s first divide our training data into a smaller training and validation set.
#split training set into training and validation setX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=4)train_errors = [] #append training errors for plot val_errors = []  #append validation errors for plot
Now let’s loop through the observations in the new training set, starting with only one observation. Based on this, we will train our linear regression model and evaluate its performance using the entire validation set. We will do this iteratively and add another observation to the training set each round, followed by a retraining of the model and another evaluation with the validation set:
#loop over each observation and iteratively compute RMSE for the new training and the entire validation set#in each iteration add one more observation to the training setfor i in range(1, len(X_train)):    linear_reg.fit(X_train[:i], y_train[:i])    y_train_predicted = linear_reg.predict(X_train[:i])    y_val_predicted = linear_reg.predict(X_val)    train_errors.append(mean_squared_error(y_train[:i], y_train_predicted))    val_errors.append(mean_squared_error(y_val, y_val_predicted))plt.plot(np.sqrt(train_errors), “r-”, linewidth=2, label=”trainining set”)plt.plot(np.sqrt(val_errors), “b-”, linewidth=2, label=”validation set”)plt.legend(loc=”upper right”, fontsize=15)plt.xlabel(“Training / Validation set size”, fontsize=15)plt.ylabel(“Root Mean Square Error”, fontsize=15)
Let us have a closer look at the produced graph and start with the RMSE on the training set (red line). Initially with only a few instances the training error is very low because the linear model can fit one or two instances very well, but once the training set starts to get bigger, the RMSE starts to rise very quickly until it reaches a constant level at about 30 instances. At this point the algorithm stopped learning what it could (remember that our data is quadratic and our model is linear) and increasing the training data size doesn’t help anymore. The validation set error (blue line) is initially very high because the linear model is only trained on a handful of instances. However it rapidly gets smaller and smaller with every new instance that is added to the training set until it quickly reaches a plateau at a slightly higher level than the training set error. These types of curves are characteristic for models that are underfitting the data.
Let us do the same exercise but by replacing the linear model with a higher degree polyonimal regression. When we look at the new curves it is hard to really tell a difference but we have to look for the subtle sings that tell us that our model is overfitting: The training error of this model is generally much lower than the training error of the previous linear model. Furthermore, for a very long time there is a larger gap between the red and blue line— as already discussed above, this is one of the main signs that your model is overfitting: A high validation error and a low training error.
A very interesting detail however can be seen on the very end of the blue line: With more training instances the validation error starts dropping significantly and ends up being quite close to the training error — resulting in a much better model in the end.
This might be maybe the most important questions and luckily I can leave you with a set of strategies that will hopefully help you building better performing models.
Use more data! As we could see on the last learning curve example, if you keep adding new features to your training data set, the validation error will eventually go down and the model will stop overfitting. Ofter however your training data set is limited so you will need a different strategy.
Regularization: In other words, put some constraints on your model and lower the degrees of freedom.The more constraints you put on your model (the fewer degrees of freedom your model has) the harder it will be to overfit the data. In our example this means lowering the polynomial degree for example. In terms of regression, you should also look into Ridge, Lasso or Elastic Net Regression which put different constraints on their model weights.
Use more data! Yep, also in this case more data will very likely boost your models performance to a certain extent. However, if your assumptions about the data are wrong and you decide to use a classifier that is not able to capture the real nature of the data, then simply adding more data wont cut it!
Increase the complexity: In the case of underfitting, putting less constraints on your model and increasing its complexity will very likely result in a better model. Increasing complexity could mean going beyond a simple linear model and making use of polynomial models, or maybe even tree based models.
Of course there is more strategies to overcome this, but data amount and model complexity can be seen as the smallest common denominator of the most of them. Before you however start switching models and changing their complexity, you should first check if your model is over- or underfitting so you can start heading into the right direction. By now you should have the necessary tools to do so in your backpack!
If you want to learn more on this, please visit the links below where I have place some excellent literature and articles on this and related topics.
[1] Géron, A., Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/
[2]Géron, A., Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow Github Code Examples: https://github.com/ageron/handson-ml2
[3] Koehrsen, W., Overfitting vs. Underfitting: A Complete Example: https://towardsdatascience.com/underfitting-and-overfitting-in-machine-learning-and-how-to-deal-with-it-6fe4a8a49dbf