Why Small Data is a Big Deal
By: Obviously Ai
Tags: Data Analysis, Artificial Intelligence, Data, AI, Data Science

Why Small Data is a Big Deal

Here’s how small data can make a big impact.

Big Data in the News

When we hear about Artificial Intelligence in the news, it’s usually about some shiny new breakthrough built using big data — things like Tesla’s self-driving cars, OpenAI’s text generators, or Neuralink’s brain-computer interfaces.

Small Data in Real Life

However, as with much of what we read in the news, these are outliers. In reality, most AI projects look completely different. Most of us aren’t trying to create superintelligence, we’re just trying to optimize KPIs, whether it’s churn, attrition, sales, traffic, or any of a million other metrics.

And to optimize KPIs, you don’t need big data. The most common sources of data for KPI optimization are day-to-day business tools like Hubspot, Salesforce, Google Analytics, or even Typeform. Unless you’re one of the outliers, the exports from these tools likely wouldn’t qualify as big data.

If you do work with big data, then all the more power to you. You’ll potentially be able to create even more accurate models, but it’s not a pre-requisite to adding value to an organization.

Small Data for Object Detection

When we hear about “big data,” it’s often in the same breath as “object detection.” Indeed, the norm for object detection models has been training on massive amounts of data.

Until now.

Researchers at the University of Waterloo released a paper discussing ‘Less Than One’-Shot Learning, or the ability for a model to accurately recognize more objects than the number of examples it was trained on.

A popular dataset for computer vision experiments is called MNIST, which is made of 60,000 training images of handwritten digits from 0 to 9.

A previous experiment by MIT researchers showed that it was possible to “distill” this huge 60,000-image dataset down to just 10 images, carefully engineered to be equal in information to the full set, achieving almost the same accuracy.

LO-Shot Learning

In the new LO-Short paper, researchers figured out they could create images that blended multiple digits together, which are fed into the model with “soft” labels, like how a man and a horse have partial features of a centaur.

In the context of MNIST, this refers to the idea of digits sharing some features, such as a digit of 3 looking somewhat like an 8, a tiny bit like a 0, but nothing like a 1 or a 7. Thus, training images can be used to understand more objects than are even in the training set.

Astonishingly, it seems there’s virtually no limit to this concept, meaning that carefully engineered soft labels could encode any number of categories.

“With two points, you can separate a thousand classes or 10,000 classes or a million classes.” (source)

More Than Theory

The paper demonstrates this concept with a kNN (k-nearest neighbors) approach, which classifies objects via separable planes along feature axes in a graphical approach. By creating tiny synthetic datasets, with carefully engineered soft labels, the kNN algorithm was able to detect more classes than there were data points.

Limitations

While this approach worked astonishingly well applied to the visual, interpretable kNN algorithm, the approach may not work for complicated and opaque neural networks.

The previous example of “data distillation” also doesn’t work as well, since it requires that you start with a very large dataset.

The Implications

More than just a fascination, this research has important implications for the AI industry, namely in reducing data requirements.

Intense data requirements make it extremely expensive to train AI models. For instance, GPT-3 cost upwards of $4 million to train. There are also concerns that inference, or making predictions with the model, is too expensive for researchers. This is more of a reflection of the extreme size and computing requirements of GPT-3, a 175-billion parameter model, than anything else.

The status quo for current State-of-the-Art models is to train on as much data as possible. GPT-3, a language model, took that to the extreme by training on essentially the entire Internet.

With the new LO-Shot Learning breakthrough, it may one day be possible to accomplish SOTA with just a few data points, resulting in extremely lightweight and efficient models.