Is your model overfitting? Or maybe underfitting? An example using a neural network
By: Serafeim Loukas
Tags: Overfitting, Neural Networks, Data Science, Underfitting, Machine Learning

4. What is Underfitting in Machine Learning

Underfitting means that our ML model can neither model the training data nor generalize to new unseen data.

A model that underfits the data will have poor performance on the training data. For example, in a scenario where someone would use a linear model to capture non-linear trends in the data, the model would underfit the data.

A textbook case of underfitting is when the model’s error on both the training and test sets (i.e. during training and testing) is very high.

5. The sweet spot and good models

It is obvious that there is a trade-off between overfitting and underfitting.

Ideally, a good model is a model that finds the sweet spot between underfitting and overfitting.

Which is the sweet spot?

The sweet spot is defined as the point where the model’s error on the training set and the test (unseen) test set remain both low. In other words, the sweet spot is the spot where the error on the test set starts to increase and, at the same time, the model’s error on the training set is acceptable (low).

6. How to (potentially) limit Overfitting

The most common problem in the ML learning filed is overfitting.

Action that could (potentially) limit overfitting:

We can use a Cross-validation (CV) scheme. Reduce the complexity of the model (make the model less complex).

When it comes to solution 1 i.e. the use of cross-validation, the most famous CV scheme is the KFolds cross-validation. Using a KFolds scheme, we train and test your model k-times on different subsets of the training data and estimate a performance metric using the test (unseen) data. Other options include the Leave-one-out cross-validation (LOOCV), the Leave-P-out cross-validation (LpOCV) and others.

KFold visually represented. Figure created by the author.

When it comes to solution 2 i.e. reducing the complexity of the model can help reduce the overfitting. For example, if someone is using an SVM model with RBF kernel then reducing the complexity would mean to use a linear kernel. In another case, if someone is fitting a polynomial to some data, then reducing the complexity would mean to fit a linear model instead (linear regression).

7. How to (potentially) limit Underfitting

In that case, there are 2 gold standard approaches:

Try another model Increase the complexity of the current model

Solution 1 is trivial. Concerning solution 2, an example an be the following: if someone is fitting a linear regression to some data, then increasing the complexity would mean to fit a polynomial model.

8. Conclusions

Overfitting or underfitting are the most common causes of poor performance for most machine learning (ML) models. Additionally, overfitting is more common than underfitting.

Overfitting is when the model’s error on the training set (i.e. during training) is very low but then, the model’s error on the test set (i.e. unseen samples) is large!

is when the model’s error on the training set (i.e. during training) is very low but then, the model’s error on the test set (i.e. unseen samples) is large! Underfitting is when the model’s error on both the training and test sets (i.e. during training and testing) is very high.

To overcome these problems, cross-validation is usually used in order to estimate the model’s performance on unseen data.

BONUS: Working example in Python

The dataset

We will use the Iris Dataset as we have done in all my previous posts. The dataset contains four features (length and width of sepals and petals) of 50 samples of three species of Iris (Iris setosa, Iris virginica and Iris versicolor).

The model

We will use a multilayer perceptron (MLP) regressor. A MLP is a class of feedforward artificial neural network (ANN). The term MLP is used ambiguously, sometimes loosely to any feedforward ANN, sometimes strictly to refer to networks composed of multiple layers of perceptrons.

from sklearn.neural_network import MLPRegressor

from sklearn.metrics import mean_absolute_error

import numpy as np

from sklearn.model_selection import validation_curve

from sklearn.datasets import load_iris

import matplotlib.pyplot as plt

from sklearn.model_selection import KFold

np.random.seed(0)

plt.style.use(‘ggplot’)

Let’s load the iris data:

iris = load_iris()

X, y = iris.data, iris.target

We will use KFolds cross-validation with 20 folds (K=20) to evaluate the generalization ability of our model. Within each fold we will estimate the training and test error using the training and test sets, respectively.

kf = KFold(n_splits=20)

list_training_error = []

list_testing_error = [] for train_index, test_index in kf.split(X):

X_train, X_test = X[train_index], X[test_index]

y_train, y_test = y[train_index], y[test_index]

model = MLPRegressor()

model.fit(X_train, y_train)

y_train_data_pred = model.predict(X_train)

y_test_data_pred = model.predict(X_test) fold_training_error = mean_absolute_error(y_train, y_train_data_pred)

fold_testing_error = mean_absolute_error(y_test, y_test_data_pred)

list_training_error.append(fold_training_error)

list_testing_error.append(fold_testing_error)

Plot the MAE of the training phase and the MAE of the testing phase: