Is your model overfitting? Or maybe underfitting? An example using a neural network | by Serafeim Loukas | Towards Data Science

Most of the times, the cause of poor performance for a machine learning (ML) model is either overfitting or underfitting. A good model should be able to generalize and overcome both the overfitting and underfitting problems. But what is overfitting? But what is underfitting? When does it mean for a model to be able to generalize the learned function/rule ?
Read on and you will be able to answer all these questions.
Generalization of a ML model refers to how well the rules/patterns/functions learned by the ML model, apply to specific examples not seen by the model when it was learning. This is usually called the unseen set or the test set.
The goal of a good ML model is to generalize well from the training data to any data that is coming from the problem’s domain. This allows the prediction of some data that the model has NEVER seen before (i.e. making prediction in the future).
Overfitting means that our ML model is modeling (has learned) the training data too well.
Formally, overfitting referes to the situation where a model learns the data but also the noise that is part of training data to the extent that it negatively impacts the performance of the model on new unseen data.
In other worlds, the noise (i.e. random fluctuations) in the training set is learned as rules/pattenrs by the model. However, these noisy learned representations do not apply to new unseen data and thus, the model’s performance (i.e. accuracy, MSE, MAE) is negatively impacted.
A textbook case of overfitting is when the model’s error on the training set (i.e. during training) is very low but then, the model’s error on the test set (i.e. unseen samples) is large!
Underfitting means that our ML model can neither model the training data nor generalize to new unseen data.
A model that underfits the data will have poor performance on the training data. For example, in a scenario where someone would use a linear model to capture non-linear trends in the data, the model would underfit the data.
A textbook case of underfitting is when the model’s error on both the training and test sets (i.e. during training and testing) is very high.
It is obvious that there is a trade-off between overfitting and underfitting.
Ideally, a good model is a model that finds the sweet spot between underfitting and overfitting.
Which is the sweet spot?
The sweet spot is defined as the point where the model’s error on the training set and the test (unseen) test set remain both low.
In other words, the sweet spot is the spot where the error on the test set starts to increase and, at the same time, the model’s error on the training set is acceptable (low).
The most common problem in the ML learning filed is overfitting.
Action that could (potentially) limit overfitting:
We can use a Cross-validation (CV) scheme.Reduce the complexity of the model (make the model less complex).
We can use a Cross-validation (CV) scheme.
Reduce the complexity of the model (make the model less complex).
When it comes to solution 1 i.e. the use of cross-validation, the most famous CV scheme is the KFolds cross-validation. Using a KFolds scheme, we train and test your model k-times on different subsets of the training data and estimate a performance metric using the test (unseen) data. Other options include the Leave-one-out cross-validation (LOOCV), the Leave-P-out cross-validation (LpOCV) and others.
When it comes to solution 2 i.e. reducing the complexity of the model can help reduce the overfitting. For example, if someone is using an SVM model with RBF kernel then reducing the complexity would mean to use a linear kernel. In another case, if someone is fitting a polynomial to some data, then reducing the complexity would mean to fit a linear model instead (linear regression).
In that case, there are 2 gold standard approaches:
Try another modelIncrease the complexity of the current model
Try another model
Increase the complexity of the current model
Solution 1 is trivial. Concerning solution 2, an example an be the following: if someone is fitting a linear regression to some data, then increasing the complexity would mean to fit a polynomial model.
Overfitting or underfitting are the most common causes of poor performance for most machine learning (ML) models. Additionally, overfitting is more common than underfitting.
Overfitting is when the model’s error on the training set (i.e. during training) is very low but then, the model’s error on the test set (i.e. unseen samples) is large!
Underfitting is when the model’s error on both the training and test sets (i.e. during training and testing) is very high.
To overcome these problems, cross-validation is usually used in order to estimate the model’s performance on unseen data.
We will use the Iris Dataset as we have done in all my previous posts. The dataset contains four features (length and width of sepals and petals) of 50 samples of three species of Iris (Iris setosa, Iris virginica and Iris versicolor).
We will use a multilayer perceptron (MLP) regressor. A MLP is a class of feedforward artificial neural network (ANN). The term MLP is used ambiguously, sometimes loosely to any feedforward ANN, sometimes strictly to refer to networks composed of multiple layers of perceptrons.
from sklearn.neural_network import MLPRegressor from sklearn.metrics import mean_absolute_errorimport numpy as npfrom sklearn.model_selection import validation_curvefrom sklearn.datasets import load_irisimport matplotlib.pyplot as pltfrom sklearn.model_selection import KFoldnp.random.seed(0)plt.style.use(‘ggplot’)
Let’s load the iris data:
iris = load_iris()X, y = iris.data, iris.target
We will use KFolds cross-validation with 20 folds (K=20) to evaluate the generalization ability of our model. Within each fold we will estimate the training and test error using the training and test sets, respectively.
kf = KFold(n_splits=20)list_training_error = []list_testing_error = []for train_index, test_index in kf.split(X):    X_train, X_test = X[train_index], X[test_index]    y_train, y_test = y[train_index], y[test_index]    model = MLPRegressor()    model.fit(X_train, y_train)    y_train_data_pred = model.predict(X_train)    y_test_data_pred = model.predict(X_test)    fold_training_error = mean_absolute_error(y_train, y_train_data_pred)     fold_testing_error = mean_absolute_error(y_test, y_test_data_pred)    list_training_error.append(fold_training_error)    list_testing_error.append(fold_testing_error)
Plot the MAE of the training phase and the MAE of the testing phase:
plt.subplot(1,2,1)plt.plot(range(1, kf.get_n_splits() + 1), np.array(list_training_error).ravel(), ‘o-’)plt.xlabel(‘number of fold’)plt.ylabel(‘training error’)plt.title(‘Training error across folds’)plt.tight_layout()plt.subplot(1,2,2)plt.plot(range(1, kf.get_n_splits() + 1), np.array(list_testing_error).ravel(), ‘o-’)plt.xlabel(‘number of fold’)plt.ylabel(‘testing error’)plt.title(‘Testing error across folds’)plt.tight_layout()plt.show()
Let’s now interpret the results and try to spot overfitting and underfitting points.
Reminder:
Overfitting is when the model’s error on the training set (i.e. during training) is very low but then, the model’s error on the test set (i.e. unseen samples) is large!
Underfitting is when the model’s error on both the training and test sets (i.e. during training and testing) is very high.
An underfitting point can be identified at fold number 10. During the 10th fold, the error on the training set and test set is at the same time high! So the model underfits the portion of the training data that are passed in for training in fold 10.
An overfitting point (not really extreme) can be seen at fold 20. The training MAE is around 0.16 and for the same fold the MAE on the test set is above 0.20.
Disclaimer: These are not extreme cases of overfitting/underfitting but I just wanted to make my point and connect it with the theory I discussed in this article.
That’s all folks ! Hope you liked this article!
If you liked and found this article useful, follow me to be able to see all my new posts.
Here is my profile. Click on Follow !
towardsdatascience.com
Questions? Post them as a comment and I will reply as soon as possible.
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
LinkedIn: https://www.linkedin.com/in/serafeim-loukas/
ResearchGate: https://www.researchgate.net/profile/Serafeim_Loukas
EPFL profile: https://people.epfl.ch/serafeim.loukas
Stack Overflow: https://stackoverflow.com/users/5025009/seralouk