Exploring SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
By: Thalles Silva
Tags: Deep Learning, Towards Data Science, Self Supervised Learning, Representation Learning, Unsupervised Learning

Unsupervised Representation Learning

Unsupervised representation learning is concerned to address the following issue:

How can we learn good representations from unlabeled data?

Besides the question of what a good representation is, learning from unlabeled data has great potential. It can unlock a number of applications that current transfer learning hasn’t been able to address. Historically, however, unsupervised representation learning has been a much harder problem than its supervised counterpart.

As a simple example, let’s consider the task of breast cancer detection. Currently, all the best solutions use ImageNet pre-trained models as a starting point in the optimization process. Interestingly, even though there is a significant difference between breast cancer slide images and regular ImageNet samples, the transfer learning assumptions still hold to some extent.

To have an idea, most supervised datasets for breast cancer detection, like the CAMELYON dataset, do not compare in size and variability with common Computer Vision supervised datasets. On the other hand, we have a massive number of non-annotated slide images of breast cancer. Thus, if we could learn good representations from the unsupervised (much larger corpora) it would certainly help to learn more specific downstream tasks that have limited annotated data.

Fortunately, visual unsupervised representation learning has shown great promise. More specifically, visual representations learned using contrastive based techniques are now reaching the same level of those learned via supervised methods — in some self-supervised benchmarks.

Let’s explore how unsupervised contrastive learning works and have a closer look at one major work on the area.

Contrastive Learning

Contrastive methods aim to learn representations by enforcing similar elements to be equal and dissimilar elements to be different.

In recent months, we have seen an explosion of unsupervised Deep Learning methods based on these principles. In fact, some self-supervised contrastive-based representations already match supervised-based features in linear classification benchmarks.

The core of contrastive learning is the Noise Contrastive Estimator (NCE) loss.

In the equation above, you can think of x+ as a data point similar to the input x. In other words, the observations x and x+ are correlated and the pair (x, x+) represents a positive example. Usually, x+ is the result of some transformation on x. This can be a geometric transform aimed to change the size, shape or orientation of x, or any type of data augmentation technique. Some examples include rotation, sheer, resize, cutout and more.

On the other hand, x- are examples dissimilar to x. The pair (x, x-) form a negative example and they are meant to be uncorrelated. Here, the NCE loss will enforce them to be different from the positive pairs. Note that for each positive pair (x,x+) we have a set of K negatives. Indeed, empirical results have shown that a large number of negatives is required to obtain good representations.

The sim(.) function is a similarity (distance) metric. It is responsible for minimizing the difference between the positives while maximizing the difference between positive and negatives. Often, sim(.) is defined in terms of dot products or cosine similarities.

Lastly, g(.) is a convolution neural network encoder. Specifically, recent contrastive learning architectures use siamese networks to learn embeddings for positive and negative examples. These embeddings are then passed as input to the contrastive loss.

In simple terms, we can think of the contrastive task as trying to identify the positive example among a bunch of negatives.

A Simple Framework for Contrastive Learning of Visual Representations — SimCLR

SimCLR uses the same principles of contrastive learning described above. In the proposed paper, the method achieves SOTA in self-supervised and semi-supervised learning benchmarks. It introduces a simple framework to learn representations from unlabeled images based on heavy data augmentation. To put it simply, SimCLR uses contrastive learning to maximize agreement between 2 augmented versions of the same image.

Credits: A Simple Framework for Contrastive Learning of Visual Representations

To understand SimCLR, let’s explore how it builds on the core components of the contrastive learning framework.

Given an input image, we create 2 correlated copies of it, by applying 2 separate data augmentation operators. The transformations include (1) random crop and resize, (2) random color distortions, and (3) random Gaussian blur.

The order of the operations is kept fixed, but since each operation has its own uncertainty, it makes the resulting views visually different. Note that since we apply 2 distinct augmentation functions on the same image, if we sample 5 images, we end up with 2 × 5 = 10 augmented observations in the batch. See the visual concept below.

To maximize the number of negatives, the idea is to pair each image (indexed i) in the batch with all other images (indexed j). Note that we avoid pairing an observation i with itself, and with its augmented version. As a result, for each image in the batch, we get 2 ×(N-1) negative pairs — where N is the batch size.