How Did Google Researchers Beat ImageNet While Using Fewer Resources?
Tags: Machine Learning, Image Classification, Supervised Learning, Research, Machine Learning Ai

How Did Google Researchers Beat ImageNet While Using Fewer Resources?

Did they invent a new model? Read on…

Image Classification algorithms have been getting better mostly by using more resources (data, computing power, and time). The best algorithms use an extra 3.5 Billion labelled images. This paper, written by Qizhe Xie et al bucks the trend. It is able to beat the current algorithms while using only an extra 300 M unlabeled images (both algorithms use the same labelled Dataset of images, on which we have the extra 3.5 B and 300 M images respectively). This means that the new method uses 12 times lesser images, while also not requiring those images to be labelled. Therefore it is many, many times cheaper while giving better results.

What does this team do differently?

An illustration from the paper showing how they trained their models

On the surface, their approach seems to be standard SSL. The key to their superior performance lies in the steps taken before and during training. The unlabeled student model is larger than the teacher. The team also injects different types of noise into the data and models, ensuring that each student learns on more types of data distributions than their teachers. This combined with the Iterative Training is especially powerful since this makes use of the constantly improving teachers.

Noisy Student uses lesser extra Data still outperforms. Looks like ResNet was ended.

All about that noise

Since we have established the importance of noise in this training, it is important to understand the different kinds of noise involved. There are two broad categories of noise that can be implemented. Model noise refers to messing with the model during the training process. This prevents overfitting and might actually boost accuracy and robustness by allowing the model to evaluate the data with different “perspectives”. The other kind is called input noise where you inject noise to the input. The researchers specifically use RandAugment to achieve this. This serves a dual purpose of increasing the variety of data, and improving the accuracy of predictions (especially for real-world data, which is very noisy).

Real Picture of the researchers in this paper

Exhibit 1: RandAugment for Input Noise

The GOAT at work

There are some really complicated noise functions in the world. RandAugment is not one of them. Don’t be fooled though, it is one of the most effective algorithms out there. It works in a really easy to understand way. Imagine there are N ways to distort an image. This can be anything, from changing some pixels to white, to shearing along an axis. RandAugment takes 2 inputs (n,m) where n is the number of distortions applied and m is the magnitude of the distortions. It returns the final image. The distortions are applied randomly, increasing the noise by adding variability.

In the image, we see RandAugment at work applying only 2 (fixed) transformations at varying magnitudes. Already these 3 samples are very different. It doesn’t take a genius to figure out how many you could get by varying both values across multiple images (answer: lots). The data augmentation should not be overlooked. It also ensures that the student model is always at least as big as the teacher, and therefore requires fewer

Exhibit 2: Dropout for Model Noise

Dropout is a process used on neural networks etc. Its process is very easy to describe: ignore some neurons every run of the network. Pictorially:

Standard DropOut Procedure

Dropout is applied because it stops overfitting. By ignoring different units, you change the output. The feedback allows it to generalize better. Additionally, dropout can be implemented to create a list of mini-learners from one network. By ensembling, the mini-learners can outperform the parent network.

The Mini-learner Ensemble to the parent network

Exhibit 3: Stochastic Depth for Model Noise

To be frank, I wasn’t super familiar with this concept. But hey that’s what Google is for. Reading into this was amazing. Probably my favorite thing this whole paper. If I had to restart life with only one memory, this would be it. Now that we have built up the hype…

Stochastic Depth involves the following steps.

Start with very deep networks During training, for each mini-batch, randomly drop a subset of layers and bypass them with the identity function. Repeat (if needed)

Since we love visualizations

This seems awfully similar to Dropout. And in a way it is. Think of it as a scaled-up version for deep networks. However, it is really effective. “ It reduces training time substantially and improves the test error significantly on almost all data sets that we used for evaluation. With stochastic depth we can increase the depth of residual networks even beyond 1200 layers and still yield meaningful improvements in test error (4.91 % on CIFAR-10).” Can’t argue with that. Will be delving into this soon. For now, look at this figure plotting error:

See that Billy? That is Peak Performance

The 3 Kinds of Noise all contribute to the training in a unique way. And all 3 are amazing for adding robustness to predictions, by adding variation to input. This is what sets it apart in robustness from all the other State of the Art models. They are in fact so effective that even without the iterative training process, the process is able to improve current state of the art networks (more details on this in Pt 2).

Just applying noise has consistent improvements.

Training Process

Now that we understand how the different steps and adjustments improve the classification, we should look into some of the implementation details.

To me, the most significant part of the training comes in step 3. The researchers state that “Specifically, in our method, the teacher produces high-quality pseudo labels by reading in clean images, while the student is required to reproduce those labels with augmented images as input.” As we have seen, adding noise to the images (or models) can drastically change how they look. By forcing the student to work with augmented images, it allows the model to predict very unclear images with great accuracy. The details of this will be in Pt 2. but for now, here’s an example of how crucial this is in predicting ambiguous images.

Take my money already, you beautiful model

Next Steps

Please leave your feedback on this article below. If this was useful to you, please share it and follow me here.

Check out my other articles on Medium. : https://rb.gy/zn1aiu

My YouTube. It’s a work in progress haha: https://rb.gy/88iwdd

Reach out to me on LinkedIn. Let’s connect: https://rb.gy/m5ok2y

My Twitter: https://twitter.com/Machine01776819

My Substack: https://devanshacc.substack.com/

If you would like to work with me email me: devanshverma425@gmail.com

Live conversations at twitch here: https://rb.gy/zlhk9y

To get updates on my content- Instagram: https://rb.gy/gmvuy9

Get a free stock on Robinhood: https://join.robinhood.com/fnud75

Here is the paper promised. Be sure to download and read it as I’ve defined important definitions and highlighted the crucial sections. It will help you understand the paper better.