Is this the end for Convolutional Neural Networks?
By: Suyash Agarwal
Tags: Convolution Neural Net, Computer Vision, Vision Transformer, Transformers, Deep Learning

Is this the end for Convolutional Neural Networks?

From handcrafted feature detectors to vision transformers, we’ve come a long way…

Photo by Agence Olloweb on Unsplash

For almost a decade, convolutional neural networks have dominated computer vision research all around the globe. However, a new method is being proposed which harnesses the power of transformers to make sense out of images. Transformers were initially designed for natural language processing tasks with primary focus on neural machine translation. The paper, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale by Alexey Dosovitskiy, Lucas Beyer et. al. from Google Research, proposes an architecture called Vision Transformer (ViT) to process image data using transformers. In this article, I will try to explain how it works.

Problem with CNNs

Before we dive into the method proposed for vision transformers, it is imperative to analyse the drawbacks and fundamental flaws in convolutional neural networks. For starters, CNNs fail to encode relative spatial information. What we mean by this is that it’s concerned with detecting certain features and does not consider their positioning with respect to each other.

Fig 1: Both images are similar for a CNN as it does not consider the relative positioning of facial features

In the above image, both of them will be classified as a face because a CNN only looks whether certain features are present in the input image and doesn’t care about their arrangement with respect to each other.

Another major flaw in CNN is that of pooling layers. Pooling layers lose a lot of valuable information such as the precise location of most active feature detector. In other words, it fails to convey the exact location of the detected feature in the image.

Transformers in Brief

Transformers in essence, use the concept of self-attention. Let’s break it down into two parts, self and attention. Attention is nothing but trainable weights that model the importance of each part of an input sentence. If suppose a sentence is input, it will look at each word of the sentence and compare its position in the sentence with respect to the position of all the words present in the same sentence (including itself) hence the term self-attention. A score is calculated based on these positional clues which is then used to encode the semantics or meaning of the sentence in a better way.

Fig 2: Screenshot taken from Tensor2Tensor implementation of Transformer

In the above example we can see that the attention unit in the transformer is comparing the position of the word ‘it’ to every other word in the sentence including itself. The different colours represent the multiple attention units working independently and simultaneously to find different patterns of these connections. Once a score is calculated by using the above comparisons, they are sent through simple feed forward neuron layers with some normalizations in the end. During training, these attention vectors are learnt.

If you want a more in-depth explanation of self attention and transformers, be sure to checkout Jay Alammar’s excellent blogpost The Illustrated Transformer or you can also checkout the original paper Attention Is All You Need by Ashish Vaswani et. al.

Model Architecture