State-Of-The-Art Image Classification Algorithm: FixEfficientNet-L2
By: Hucker Marius
Tags: Algorithms, Efficientnet, Computer Vision, Image Classification, Data Science

State-Of-The-Art Image Classification Algorithm: FixEfficientNet-L2

FixEfficientNet is a technique combining two existing techniques: The FixRes from the Facebook AI Team[2] and the EfficientNet [3] first presented from the Google AI Research Team. FixRes is the short form for Fix Resolution and tries to keep a fixed size for either the RoC (Region of Classification) used for train time or the crop used for test time. The EfficientNet is a compound scaling of the dimensions of a CNN which improves both accuracy and efficiency. This article is meant to explain both techniques and why they are state-of-the-art.

The FixEfficientNet has been presented first with the corresponding paper on the 20th April 2020 from the Facebook AI Research Team [1]. The technique is used for Image Classification and consecutively a task of the field of Computer Vision. It is currently the state-of-the-art and has the best results on the ImageNet Dataset with 480M params, a top-1 accuracy of 88.5%, and top-5 accuracy of 98,7%.

But let’s dive in a bit deeper to get a better understanding of the combined techniques:

Understanding FixRes

Training Time

Until the Facebook AI Research Team proposed the FixRes technique the state-of-the-art was to extract a random square of pixels out of an image. This was used as RoC for the training time. (Be aware that using this technique the amount of data is artificially increased). The image has then been resized to obtain an image of a fixed size (=crop). This was then fed to the Convolutional Neural Network [2].

RoC = rectangle/square in input image

crop = pixels of RoC rescaled with a biliniear interpolation to a certain resolution

Train-time scale augmentation

TTo get a better understanding of what FixRes exactly does let’s take a look at the math. Changing the size of the RoC in the input image affects the distribution of the size of the object given to CNN. The object has a size of r x r in the input image. If the RoC is now scaled it is changed by s and consecutively the size of the object is now rs x rs.

For the augmentation, the RandomResizedCrop of PyTorch is used. The input image has a size of H x W, from which a RoC is randomly selected. This RoC is then resized to a crop of size

source: image of the author.

The scaling of the input image (H x W) to the crop that is output can be expressed by the following factor:

source: image of the author.

Test time

At test time the RoC is often centered in the image, which results in a so-called center crop. Both crops, the one from train time and test time have the same size, but they origin from a different part of the image. This often leads to a bias in the distribution of CNN [2].

Test-time scale augmentation.

As previously described the test augmentation is not the same as the training time augmentation (keyword center crop). The crop has then the size

Regarding the assumption that the input image is a square (H=W) the scaling factor for the test augmentation can be expressed as

What’s the finding of this?

Until the FixRes was developed, the preprocessing for test and training time was developed separated from each other leading to bias. Consecutively the Facebook AI Team tried to find a solution that executes the preprocessing simultaneously and somehow synchronized and that is FixRes.

source: image of the author.

The Standard Preprocessing as seen above often enlarges the RoC at training and decreases the size at test time.

The FixRes technique takes an either-or approach. It either reduces the train-time resolution and keeps the size of the test crop or increases the test-time resolution and keeps the size of the training crop. The aim is to retrieve the object (here the crow) in the same size to reduce the scale invariance in the CNN [2]. This looks like the following:

source: image of the author.

This results in two effects on how the data is fed to the CNN:

The size of the object (here the crow) in the image is changed by the FixRes Scaling. The use of different crop sizes has an impact on how and when the neurons are activated.

The Problem of Varying Activation Statistics

Touvron et al. found that larger test crops and foremost the adjustment of the object size lead to better accuracy. However, it’s a trade-off between adjusting the size of the object and varying activation statistics.

Tests showed that the activation map changes with the resolution of the image. K_test = 224 leads to a map of 7x7, K_test = 64 leads to a map of 2x2 and K_test = 448 leads to a map of 14x14. This shows that the activation distribution varies at test time and the values are out of the classifier range [1].

To solve the problem of changes in the activation statistics there are two solutions presented:

Parametric Adaptation: A parametric Fréchet distribution is used to fit the average pooling layer. The new distribution is then mapped via scalar transformation to the old distribution and applied as an activation function. Fine-tuning: Another way to apply a correction is the fine-tuning of the model. The Fine-Tuning is only applied to the last layers of the CNN.

During the fine-tuning stage, a label smoothing is used [1].

EfficientNet Architecture [3]

The authors have pre-trained several models from which the EfficientNet-L2 shows the best results. But what is the EfficientNet?

As most algorithms in Image Classification, the Efficient Net is based on CNNs. If you don’t have a clue what a CNN is, click here. CNN has three dimensions: width, depth, and resolution. Depth is the number of layers, width is the number of channels (e.g. conventional RGB would have 3 channels) and the resolution is the pixels of an image.

EfficientNets introduced compound scaling, which makes use of all three dimensions:

Width Scaling — The width can be increased by having images with more channels. The accuracy gain diminishes pretty quickly though.

Depth Scaling — Is the conventional and most typical way of scaling. By increasing depth, you increase the number of layers your neural network has. But adding more layers does not always increase the performance of your network. Most often it needs more time, but due to vanishing gradients, the performance can stagnate or even decrease with a higher number of layers.

Resolution Scaling — This means increasing the resolution and hence, the number of pixels, e.g. from 200x200 to 600x600. The problem with this kind of scaling is that the accuracy gain disappears with a higher resolution. Until a certain point, your accuracy might increase but the accuracy increments decrease.

The upscaling of all three dimensions leads to diminishing accuracy increments and a balanced scaling of all three dimensions is necessary to achieve the best accuracy results. Therefore the compound scaling is proposed:

source: image of the author.

ɸ specifies the available resources, while alpha, beta, and gamma are responsible for the allocation of them.

Touvron et al. [1] “use neural architecture search to develop a new baseline network, and scale it up to obtain a family of models, called EfficientNets.” The Neural Architecture Search (NAS) optimizes FLOPS and accuracy.

Conclusion

The combination of both techniques leads to the currently best algorithm in image classification close ahead of EfficientNet Noisy Student. It is the current leading algorithm in both, efficiency and accuracy. Due to its top-5 accuracy of 98,7% there is still the possibility for improvement, but it’s quite accurate already. So it remains to wait until this is replaced by a new technique.

Since this article does not include any implementation, you can try it by yourself using the official Github of the authors: http://github.com/facebookresearch/FixRes.

The pre-trained networks of the authors [1] can be seen below.

Screenshot of the github repo.

I hope you understood and enjoyed it!