Why Overfitting is a Bad Idea and How to Avoid It (Part 1: Overfitting in General)
By: Andrew R. Freed
Tags: Methodology, Testing, Machine Learning, Artificial Intelligence, Generalization

Beware of overfitting! Photo by Jenny Hill on Unsplash

We want our AI models to be as accurate as they can be. That’s one of the selling points of AI — that we can encode the best version of our past knowledge and have an automated model infer and apply our judgement. How can we tell when the model is accurate enough to trust? More importantly how can we tell if our efforts to improve accuracy are actually making the model worse? This situation can happen through a training problem called overfitting.

In this two-part series I will show how overfitting can affect various kinds of AI models. In this first post I’ll take a simple example and demonstrate both a reasonably accurate simple model as well as a “perfectly accurate” (overfit!) complex model. In the second post I will demonstrate overfitting in virtual assistants (sometimes called chatbots).

Modeling some sample data

Let’s assume we are trying to predict the sale price of a house. In this hypothetical example we have only two features available to us: the square footage and the sale price of the house. Here’s the sample data:

Square Footage Sale Price 500 60000 1000 121000 1250 128750 1500 175500 2000 208000

Given the limited amount of data available we opt for a simple model first. With two variables our simplest model is a purely linear regression.

Linear plot of square footage vs sales price

As we see in the image above this model looks great! The predicted model is “y = 99.7x + 14025” with an R-squared value of 0.97. (Spoiler alert: This sample data was generated with “y=100x + RANDOM_NOISE”, so this is in fact the “correct” model). While we don’t have any blind data to test our model on, we can glance at that chart and infer the model has captured the essence of the square footage/sale price relationship. We deliver our findings including the model and its 97% accuracy score.

100% or bust!

Let’s assume a distressing turn of events. Our executives are upset at the “low” accuracy and further that our model did not correctly predict any of the training examples. We are further told that only 100% accuracy is acceptable and must go back to modeling.

The training accuracy of a model can almost always be increased by adding additional features. Even though we have only one obvious input feature (x=square footage), we can generate additional features through a process called “feature engineering”. In this example we will use additional polynomial factors as our additional features (x, x², x³, x⁴).

Polynomial regression applied to square footage vs sales price

This new model “y = -6E-07x4 + 0.003x3–5.2477x2 + 3868x — 900000” has an R-squared score of 1.0 and it perfectly predicts each of the training data points. (In fact we can always achieve 100% accuracy while testing new data by adding even more polynomial features.). However this model makes several troubling implications:

· The most expensive possible house is 1879 square feet — all larger houses are cheaper than that

· Sufficiently large (or small) homes have negative cost(!)

We can compare the two models by selecting data points not in our training set and comparing the predictions.

Square Footage Price (Linear Model) Price (Polynomial Model) 418 55700 0 1750 188500 248700 1879 201361 263195 2270 240344 0 2500 263275 -590625 3000 313125 -4125300

There are other techniques we could use to show that the complex model is a much worse model despite its high training accuracy. (The most common method is cross-validation where a subset of the training data is removed and used to test the model instead.)

Generalization is the key

Rather than obsessing over training accuracy we should focus on how the model performs on data it has not seen before. This is best measured with a blind test, where we have real-world data the model was not trained on but can be tested on. In situations where no blind data is available, we can still look for ways to infer how the model works — in this case a simple visualization was enough.

As demonstrated above increasing the training accuracy of a model is not always a good thing, especially when it forces the model to learn localized but not generalizable patterns. A simpler model with lower training accuracy can be superior to a more complex model with higher training accuracy particularly when tested on unseen data.

Training accuracy is interesting but blind test accuracy is most important.

This post was intentionally simplistic in the model and data selected to demonstrate a framework for thinking about and visualizing overfitting. While very few AI models work in two dimensions this framework is still “close enough” to reasoning about more complex models. In the next post I will explore overfitting in virtual assistants.

For help in implementing best practices, and avoiding overfitting, reach out to IBM Data and AI Expert Labs and Learning.