Real Artificial Intelligence: Understanding Extrapolation vs Generalization
By: Andre Ye
Tags: Machine Learning, Data Science, Artificial Intelligence, AI, Data Analysis

Source. Image free to share.

Real Artificial Intelligence: Understanding Extrapolation vs Generalization

Stop confusing the two

Machine learning models don’t need to be intelligent — most of their applications entail performing tasks like recommending YouTube videos or predicting a customer’s next move. It is important to understand the difference between extrapolation and generalization/interpolation to understand what it really means for a model to be intelligent and to avoid a common issue of confusing the two, which is often the root cause of the implementation failures of many models.

Generalization is the entire point of machine learning. Trained to solve one problem, the model attempts to utilize the patterns learned from that task to solve the same task, with slight variations. In analogy, consider a child being taught how to perform single-digit addition. Generalization is the act of performing tasks of the same difficulty and nature. This may also be referred to as interpolation, although generalization is a more commonly used and understood term.

Created by Author.

Extrapolation, on the other hand, is when the model is able to obtain higher-dimensional insights from a lower-dimensional training. For instance, consider a first grader who is taught single digit addition, then presented with a multi-digit addition problem. The first grader thinks, “okay, so when the units digit adds to larger than ten, there is a tens component and a ones component. I take that into account and add a one to the tens column to account for that.” This is, of course, the key insight around which arithmetic is based around, and if you thought hard enough about what it means to add and understood place value, you could figure it out. Yet most first graders never realize this, rarely discovering how to carry over on their own.

Created by Author.

It’s important to realize that extrapolation is hard. Even many humans cannot succeed at extrapolation — indeed, intelligence really is a measure of being able to extrapolate, or to take concepts explained in a lower dimension and being able to apply them at a higher one (of course, dimension as in levels of complexity, not literally). Most IQ tests are based around this premise: using standard concepts in ways only a true extrapolator could understand.

In terms of machine learning, one example of extrapolation can be thought of as being trained on a certain range of data and being able to predict on a different range of data. This may be easy with simple patterns, such as simple positive number/negative number or in a circle/not in a circle classifications.

Created by Author.

Yet the ability for models to extrapolate on more complicated patterns is limited with traditional machine learning methods. Consider, for example, the checkerboard problem, in which alternating squares on a two-dimensional plane are colored as either 0 or 1. To a human, the relationship is clear and one could go on coloring an infinite checkerboard, given only the rules of a smaller, finite x by x checkerboard. While the checkerboard problem is defined by a set of rules for humans (no two squares who share a side can be the same color), mathematically it is defined as such:

Created by Author.

This is a very unintuitive way of thinking, and most models do not think to generate rigorously mathematical, extrapolative definitions. Instead, regular algorithms essentially attempt to split the feature space geometrically, which may or may not work, given the task. Unfortunately, in this case, they do not extrapolate to coordinates outside of the coordinates that they were trained on. Even many neural networks fail to perform at this task.

Results of KNN & Decision Tree when trained on 20x20 checkerboard and told to predict 40x40 space. Created by Author.

Sometimes, however, extrapolation is not as much about recognizing complex relationships as it is finding a smart, extrapolate-able solution to carrying out the task on foreign ranges. For instance, the checkerboard problem can be solved by drawing diagonal lines. This solution has been observed to be discovered with, for example, neural network ensembles.

Neural Network Result from “Competitive Learning Neural Network Ensemble Weighted by Predicted Performance”. Created by Author.

A common argument you’ll hear in the debate of machine intelligence is that “machines can only do one thing really well.” This is indeed the definition of interpolation or generalization — to perform tasks inter, or within, a predefined set of rules. Yet extrapolating requires having such a solid understand of the concepts inter that they can be applied extra, or outside the taught region. Rarely can current machine learning models extrapolate reliably; usually, the ones that show promise are geometrically easier to extrapolate one and fail at other problems. All artificial intelligence methods are interpolative by nature, and it’s debatable if it is even possible to artificially construct an extrapolative (“intelligent”) algorithm.

Extrapolation is seldom the goal of modelling or machine learning, but often it is used interchangeably with generalization — the most obvious, of course, being linear regression, in which taking its infinity-tending predictions as gold is more common and less noticeable when there are multiple dimensions at play (multiple regression).

Another example of extrapolation is when, say, companies will train a model on outlier-free data and then implement it in real life, in which outliers are much more abundant and cannot simply be ignored. This is common in implementations of models that often goes undetected but may be a big reason why your model may not be performing up to test results in real life. When there is a disparity between the data on which the model was trained on and the data the model is expected to predict on, it is likely that you may be asking the model to extrapolate.

XKCD. Sometimes even humans are bad extrapolators. Image free to share.

Models generally cannot extrapolate well, be it in a measure of symbolic intelligence or in real applications. It’s important to make sure that your model is not being confronted with an extrapolation task — current algorithms, even as complex and powerful as neural networks, are simply not designed to perform generally well on extrapolation tasks. A decent check to check for extrapolative tasks is to plot out the distributions of each column in the training and testing set, then see if the testing set is significantly non-compatible with the training one.

Until we can create a machine learning algorithm that is capable of extrapolating generally across all problems, similarly to how the concept of a neural network (of course, with some varying architectures) can single-handedly address almost any generalization problem, they will never truly be able to be ‘intelligent’ and perform tasks outside the narrow scope in which they are trained on.