Less is More: Labeled data just isn’t as important anymore | by Abhinav Raghunathan | Towards Data Science

I used to always think of data as being inherently calm and ordered — a neatly packaged array of information ready to process. I think most people who haven’t had a taste of the chaos of the real world would believe the same. Professionals (or, really, anyone who has worked with data) know that data is far more fluid and amorphous. Very rarely is data ordered and structured with a nice label to go with it. In fact,
most of the time, data is unlabeled, unstructured, and chaotic.
Unfortunately, most of the machine learning algorithms that are widely used today depend heavily on labeled data and fully supervised algorithms. Data scientists and data engineers spend a lot of time and effort fighting entropy and producing these clean datasets that we’re so used to seeing on websites like Kaggle.
It turns out, however, that a semi-supervised approach with deliberate noise injection might just be better than any supervised learning approach, even while using significantly less labeled data.
Supervised learning requires a lot of labeled data that most of us just don’t have. Semi-supervised learning (SSL) is a different method that combines unsupervised and supervised learning to take advantage of the strengths of each. Here’s one possible procedure (called SSL with “domain-relevance data filtering”):
1. Train a model (M) on labeled data (X) and the true labels (Y).2. Calculate the error.3. Apply M on unlabeled data (X’) to “predict” the labels (Y’).4. Take any high-confidence guesses from (2) and move them from X’ to X.5. Repeat.
Over time, the labeled dataset grows in size and affords the model a fair bit of data to work with. This is already impressive because unlabeled data is far cheaper and more abundant than labeled data, but SSL isn’t guaranteed to be a good solution (Singh, Nowak, & Zhu). There is, however, a new study that suggests that there are a couple things we can change to boost performance of SSL.
What if I told you this new version of SSL almost outperforms supervised learning with only a fraction of the necessary labeled data?
If your kids learned exclusively from what you tell them, chances are that there would be a lot of situations they would be completely lost in. Machine learning algorithms are the same way. It turns out, giving algorithms tough pieces of data to work with make them significantly more robust later on. These “tough pieces of data” can also be called “noise.” A really useful technique for strengthening machine learning algorithms is intentionally adding noise (called “noise injection” or “data augmentation” in this paper) to data. In a new paper by CMU and the Google Brain Team, a new SSL framework that cares about “quality of noise” injected as well. Here’s the overarching model as a flowchart:
Here’s the same flowchart in a step-by-step format:
1. Train a model (M) on labeled data (X) and true labels (Y).2. Calculate the supervised error.3. Apply M on unlabeled data (X’) to “predict” the labels (Y’).4. Introduce noise to X’ in a smart way to produce X’’.5. Apply M on unlabeled, perturbed data (X’’) to “predict” the labels (Y’’).6. Calculate the unsupervised error by comparing Y’ and Y’’.7. Calculate the total error using (2) and (6).8. Take any high-confidence guesses from (3) and move them from X’ to X.9. Repeat.
There are lots of ways to do (4) in the above step-process. One interesting method is called back-translation. Consider the sentence:
“Labeled data is not as cheap as unlabeled data and therefore is a relic of the past.”
I can convert this sentence into another language and then back into English again. Here’s the result when I use Norwegian as my intermediate.
“Marked data is not as cheap as unmarked data and is therefore a thing of the past.”
Now we’ve added noise. Subtle changes, but a world of difference to an algorithm.
The paper mentioned earlier does a stress-test of the approach outlined in the last section. Using Google’s BERT (Large) algorithm on text classification datasets, they’ve generated the following results, where n is the number of labeled samples used):
       + — — — — + — — — — — — — —  + — — — — — — — — — — — +       | Dataset | Supervised Error | Semi-Supervised Error |       + — — — — + — — — — — — — —  + — — — — — — — — — — — +       | IMDb    | 4.51  (n = 20k)  | 4.78 (n = 20)         |       | Yelp-2  | 1.89  (n = 560k) | 2.50 (n = 20)         |       | Yelp-5  | 29.32 (n = 650k) | 33.54(n = 2.5k)       |       | Amazon-2| 2.63  (n = 3.6m) | 3.93 (n = 20)         |       | Amazon-5| 34.17 (n = 3.0m) | 37.80(n = 2.5k)       |       | DBpedia | 0.64  (n = 560k) | 1.09 (n = 140)        |       + — — — — + — — — — — — — —  + — — — — — — — — — — — +
These results imply that this particular SSL method works almost as well with just a small fraction of the labeled data as supervised methods do with ALL of the labeled data!
As for the usefulness of labeled data — naturally, if labeled data is available, you take it. But with results like these, any rationale for creating neat sets of order, structured, labeled data is rapidly vanishing, especially because unlabeled data is widely available and extremely cheap (computation-, time-, and effort-wise). The future is unpredictable (and unlabeled), but we’re slowly gaining the tools necessary to capture it!
The original version of this post can be found here.