Transfer learning: the dos and don’ts
By: Chris Von Csefalvay
Tags: Artificial Intelligence, Neural Networks, AI, Deep Learning, Machine Learning

If you have recently started doing work in deep learning, especially image recognition, you might have seen the abundance of blog posts all over the internet, promising to teach you how to build a world-class image classifier in a dozen or fewer lines and just a few minutes on a modern GPU. What’s shocking is not the promise but the fact that most of these tutorials end up delivering on it. How is that possible? To those trained in ‘conventional’ machine learning techniques, the very idea that a model developed for one data set could simply be applied to a different one sounds absurd.

The answer is, of course, transfer learning, one of the most fascinating features of deep neural networks. In this post, we’ll first look at what transfer learning is, when it will work, when it might work, and why it won’t work in some cases, finally concluding with some pointers at best practices for transfer learning.

What is transfer learning?

Think of transfer learning like a container ship that takes the weights from one model and deploys them in your use case.

In their seminal paper on the subject, Pan and Yang (2010) give an elegant mathematical definition (see Subsection 2.2) of transfer learning. For our purposes, however, a much simpler definition will suffice: transfer learning uses what was learned for a particular task (sometimes called the source task) to solve a different task (the destination task). The assumption there is, of course, that the source and the destination task are sufficiently similar. That assumption is at the heart of transfer learning. We must therefore first understand why transfer learning works in the first place.

You might recall that your typical contemporary neural network — say, a multi-layer convolutional deep neural network (cDNN)–– consists of layers of neurons, each feeding forward its result according to the weights and biases of the network that were calculated as a result of training. This architecture was patterned after a particular process in the human brain that is rather similar to this, namely the Ventral Visual Stream.

An outline of the Ventral Visual Stream. The optic nerve (dark purple) carries information from both sides of the retina to the optic chiasm (OX), where they are separated depending on whether they came from the left or right part of the visual field. The lateral geniculate nucleus (LGN) separates the type of input. The projections from the LGN each side go into the corresponding primary visual area (V1). From there, it traverses the Ventral Visual Stream in increasing complexity and semanticity. Author’s figure.

The ventral stream begins in the primary visual cortex or V1, a part of the occipital lobe. V1 receives visual information from the lateral geniculate nucleus, a part of the thalamus that sends optical information to the occipital lobe while segmenting it into two types of information — inputs that are more useful for determining what you’re looking at (parvocellular layer of the LGN) and inputs more useful for determining where something is (magnocellular layer of the LGN). Both end up in the primary visual cortex, but the ventral visual stream is fed by the ‘what’ information, from the parvocellular layer of the LGN: slow, but sustained and detailed information about objects, originating from retinal ganglion cells called P cells.

The outputs (sometimes called projections) of P cells are then processed as they traverse the ventral visual stream, from V1 through V2, V4 and eventually the inferior temporal (IT) lobe. It was found using excitation experiments that while cells in V1 respond to pretty simple patterns (primarily, edges, their orientation, colours and spatial frequencies), V2 responds to more complex notions, including certain Gestalt phenomena such as ‘subjective contours’. Eventually, proceeding gradually to the inferior temporal lobe, neurons respond to increasingly complex patterns.

CaffeNet’s cat recogniser, reverse engineered by DeconvNet. Note the increasing semanticity as the layer number increases, from simple geometries in convolutional layer (CL) 1 to geometries in context (CL 2), composite geometries (CL 3 and 4) to semantically meaningful and specific filters for the class of objects (CL 5). Image from Qin et al. (2018).

We see this replicated in deep neural networks. A technique called a Zeiler-Fergus deconvolutional network allows us to see what patterns a particular layer (often referred to as a ‘filter’ in this context) would most respond to (activation maximisation)— that is, the ‘kinds’ of structures that excite that particular neuron the most (you might be familiar with some of these shapes from Google’s DeepDream algorithms). As the layers increase, the complexity of shapes increases. While we see simple edges or even blocks of colours in the first layers, later layers reveal more complex patterns and final layers’ activations are often recognisable as the intended class. They also become increasingly semantic — a combination of lines are together weighted as a filter that recognises a triangle, combinations of triangle filters recognise two ears, in combination with other filters, it begins to recognise a dog’s face as distinct from a cat’s.

The idea of transfer learning is inherent in the fact that neural networks are layer-wise self-contained — that is, you can remove all layers after a particular layer, bolt on a fully connected layer with a different number of neurons and random weights, and get a working neural network. This is the basis of transfer learning. In transfer learning, we use what well-trained, well-constructed networks have learned over large sets, and apply them to boost the performance of a detector on a smaller (usually by several orders of magnitude!) data set.

To transfer or not to transfer

From the above, some facts emerge about the utility (and disutility) of transfer learning. The biggest benefit of transfer learning shows when the target data set is relatively small. In many of these cases, the model may be prone to overfitting, and data augmentation may not always solve the overall problem. Therefore, transfer learning is best applied where the source task’s model has been trained on a vastly bigger training set than could be acquired for the destination task. This may be the case because instances for a particular thing are hard to come by (e.g. when working on the synthesis or recognition of a voice of which only few samples exist), or where labelled instances are difficult to obtain (e.g. in the context of diagnostic radiology, labelled images are often hard to get, especially for rare conditions).

A model where there are approximately the same amount of data for each task might still benefit from transfer learning if there is a risk of overfitting, as it often occurs when the destination task is highly domain-specific. In fact, in training a large domain specific dCNN might be counterproductive, as it might overfit to the particular domain. It is overall then sometimes advisable to use transfer learning where the source and destination task’s training sets are of the same size.

In practice, in computer vision, it is very common to use gold standard networks, trained on massive image data sets––such as ImageNet’s 1.2 million images over 1,000 categories––as the point of departure for even quite domain-specific tasks, such as evaluating chest radiographs. A number of modern machine-learning packages, especially high-level packages like Caffe , fastai and keras , come with their own built-in model zoo, allowing easy access to a pre-trained dCNN. It is, then, only a question of removing the topmost layers, adding one or more new layers and retraining (fine-tuning) the model. The new model will immediately benefit for the weeks of laborious training that go into creating a model like ResNet50 , NASNet or Inceptionv3 . Overall, when used appropriately, transfer learning will give you a trifecta of benefits: a higher starting accuracy, faster convergence and higher asymptotic accuracy (the accuracy level to which the training converges). Recently, some fairly promising sites have sprung up that catalogue various pre-trained models––my favourites are ModelDepot and ModelZoo, the latter having a vast database filterable by framework and solution, including many pre-trained GANs.

Some best practices