Transfer Learning in Action: From ImageNet to Tiny-ImageNet | by Thushan Ganegedara | Towards Data Science

Transfer learning is an important topic. As a civilization, we have been passing on the knowledge from one generation to the other, enabling the technological advancement that we enjoy today. It’s the edifice that supports most of the state-of-the-art models that are blowing steam, empowering many services that we take for granted.
Transfer learning is about having a good starting point for the downstream task we’re interested in solving.
In this article, we’re going to discuss how to piggyback on transfer learning to get a warm start to solve an image classification task. The content of this article is based on “TensorFlow 2 in Action” by Manning and on TensorFlow 2.2.
Code for this exercise: Here.
In this tutorial we’re going to cover,
What is Transfer Learning?
Understanding the dataset
Learning the nitty-gritties of InceptionNet-Resnet-v2
Download a ImageNet pretrained InceptionNet-Resnet-v2 and use it to learn the classification task (Tiny-ImageNet) at hand quickly
Transfer learning itself is a very simple yet powerful concept. It is about solving a bespoke task by leveraging the knowledge impregnated in a model that’s already good at solving a similar task. Following the long-standing tradition of “a picture worth a thousand words”, here’s a depiction of transfer learning process.
For example,
if you develop a segmentation model for self-driving cars and you use a pretrained image classification model to start with or,
if you download BERT and build a spam classifier on top of features provided by BERT,
that is essentially transfer learning. However to understand the idiosyncratic methods that enables us achieve transfer learning will take time. Different ways you achieve transfer learning through, will have different advantages or effects. If you are interested, here’s a good guide/portfolio of different methods.
The dataset we’re going to use is Tiny-ImageNet. It is a smaller version derived from the monolith ImageNet challenge. The dataset is pretty straight-forward. There’s 100,000 training and 10,000 validation (will be used as the testing data) samples. Then each record has,
A RGB image of size 64x64x3
A label indicating the object present in the image
Here’s a few examples from the dataset.
I’m going to skim through the details of data extraction. However, if you are interested in the details, please refer the code. To feed this data we will first download the dataset (the code is provided). It has two datasets; training data and testing data. We will split the train dataset to two subsets,
Training data
Validation data
Note that the testing data is called “validation” data when you download it, as the Tiny-ImageNet has an undisclosed test dataset that. But we’re going to consider that as the test set. First we will define two tensorflow.keras.preprocessing.image.ImageDataGenerator objects.
# For train and validation dataimage_gen_aug = ImageDataGenerator(                                                   samplewise_center=False,                                                    rotation_range=30,                                                         width_shift_range=0.2, height_shift_range=0.2,                                brightness_range=(0.5,1.5),                                                 shear_range=5,                                                              zoom_range=0.2,                                                             horizontal_flip=True,                                                        fill_mode='reflect',                                                        validation_split=0.1                                                  )# For the testing data (No augmentation)image_gen = ImageDataGenerator(samplewise_center=False)
We’re going to augment the training data so that we can reach a higher accuracy than using the original data as it is. Data augmentation is when you create different variants of your inputs (e.g. change brightness/contrast, translate, rotate, etc.) so that you generate more data for your model. You can learn more about data augmentation in Chapter 7 of TensorFlow 2 in Action by Manning. Another important observation is how we use the validation_split argument to take out a chunk of data (10%) as validation data. Then we are going to use,
# Get training data generatortrain_gen = image_gen_aug.flow_from_directory(..., subset='training')# Get valid data generatorvalid_gen = image_gen_aug.flow_from_directory(..., subset='validation')# Get test data generatortest_gen = image_gen.flow_from_dataframe(...)
to obtain three data generators (i.e. Python generators); training, validation and testing data generators. Note how samplewise_center is set to False. It is a normalization step that is commonly used for image process. I’ve set it to False because I’m going to introduce few more custom augmentation steps as follows.
def data_gen_augmented(gen, random_gamma=False, random_occlude=False):    for x,y in gen:         if random_gamma:            # Gamma correction            # Doing this in the image process fn doesn't help improve performance            rand_gamma = np.random.uniform(0.9, 1.08, (x.shape[0],1,1,1))            x = x**rand_gamma                    if random_occlude:            # Randomly occluding sections in the image            occ_size = 10            occ_h, occ_w = np.random.randint(0, x.shape[0]-occ_size), np.random.randint(0, x.shape[0]-occ_size)            x[::2,occ_h:occ_h+occ_size,occ_w:occ_w+occ_size,:] = np.random.choice([0.,128.,255.])                # Image centering        x -= np.mean(x, axis=(1,2,3), keepdims=True)                # Making sure we replicate the target (y) three times        yield x,(y,y,y)
Basically, we do,
Random Gamma correction
Introduce random occlusions in the images
Then samplewise_center is performed manually with the line,
x -= np.mean(x, axis=(1,2,3), keepdims=True)
in the above code. Finally, we define the data generators that uses both built-in data augmentation steps as well as the custom augmentation steps.
train_gen_aux = data_gen_augmented_inception_resnet_v2(train_gen, random_gamma=True, random_occlude=True)# We do not augment data in the validation/test datasetsvalid_gen_aux = data_gen_augmented_inception_resnet_v2(valid_gen)test_gen_aux = data_gen_augmented_inception_resnet_v2(test_gen)
These data generators will be used to fit the model once it’s defined. Again if you would like to see the full code related to data download and creating training/validation/testing data pipelines, refer to the code.
At the first glance, Inception-Resnet-v2 architecture might look like the Pandora box. Because Inception-Resnet-v2 evolved to become what it is, while going through many iterations. Specifically, Inception net has the following models.
Inception-Net-v1 (Paper)
Inception-Net-v2 (Paper)
Inception-Net-v3 (Paper)
Inception-Net-v4 (Paper)
Inception-Resnet-v1 (Paper)
Inception-Resnet-v2 (Paper)
Xception (Paper)
If you would like more information about these different models, the motivation behind the design principles they are built upon, read Chapter 6 of TensorFlow 2 in Action by Manning. The Inception-Resnet-v2 model has the following building blocks.
A stem (White) — Stem stands as the back bone of the image classification model. It is followed by several repeated convolution blocks and then a prediction layer.
Inception Resnet blocks (Type A/B) (Red) — These is the most notable contribution of Inception Resnet-v2. These blocks uses multiple streams of convolution paths as well as residual connections.
Reduction blocks (Green) — Primary purpose of these blocks is to reduce the height and width dimensions in a controlled fashion.
Final prediction layer (Blue)— Outputs the probability distribution over the available classes for a given input/ batch of inputs.
Now it’s important to understand that InceptionNet models are not just about a random configuration of a CNN that works! They are actually based on some strong foundational principles.
Inception Net models are designed to make CNNs parameter efficient, without sacrificing performance too much
Introducing sparsity (i.e. less parameters) within the CNN by using different kernel sizes in convolution layers
Factorizing large convolution filters to smaller ones
Using residual connections and batch normalization to make the model training smooth
If you would like to learn these concepts in detail, Chapter 6 and 7 of “TensorFlow 2 in Action” by Manning goes into great lengths explaining these concepts.
Here we’re going to download the pretrained Inception-Resnet-v2 model. You can download this model through the sub-API tensorflow.keras.applications .
from tensorflow.keras.applications import InceptionResNetV2from tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import Input, Dense, DropoutK.clear_session()# We're going to download the InceptionResNetv2 model, remove the top layer and wrap it around an input layer and a prediction layer (with 200 classes)model = Sequential([    Input(shape=(224,224,3)),    InceptionResNetV2(include_top=False, pooling='avg'),    Dropout(0.4),    Dense(200, activation='softmax')])# Defining the lossloss = tf.keras.losses.CategoricalCrossentropy()# Defining the optimizeradam = tf.keras.optimizers.Adam(learning_rate=0.0001)# Compiling the modelmodel.compile(loss=loss, optimizer=adam, metrics=['accuracy'])model.summary()
Here, we are going to create a tf.keras.models.Model by wrapping,
An Input layer having the size 224x224x3
the downloaded Inception Resnet v2 model (without the top prediction layer and an average pooling layer on top)
A dropout layer with 40% dropout rate
A final prediction layer
Note that the input layer expects a 224x224x3 sized image. However the images we have are of size 64x64x3. To solve this problem, we can easily resize the images through the ImageDataGenerator. We’re also using a smaller learning rate (Adam has a default learning rate of 0.001). This is because a higher learning rate is unnecessary (as we’re not starting out with a random initialization) and can even corrupt the knowledge embedded in the model.
Now the fun begins! We’re about to train the model.
from tensorflow.keras.callbacks import EarlyStopping, CSVLogger, ReduceLROnPlateau# Defining callbackses_callback = EarlyStopping(monitor='val_loss', patience=10)csv_logger = CSVLogger(os.path.join('eval','4_eval_resnet_pretrained.log'))n_epochs=50lr_callback = ReduceLROnPlateau(    monitor='val_loss', factor=0.1, patience=5, verbose=1, mode='auto')# Training the modelhistory = model.fit(    train_gen_aux,     validation_data=valid_gen_aux,     steps_per_epoch=get_steps_per_epoch(int(0.9*(500*200)), batch_size),     validation_steps=get_steps_per_epoch(int(0.1*(500*200)), batch_size),    epochs=n_epochs,     callbacks=[es_callback, csv_logger, lr_callback])
There’s a lot going on here. Let’s break things down and understand what’s going on here. First we are defining a bunch of callbacks. These callbacks, when passed to the model.fit(...) are called at the end of the epoch. For example,
es_callback — Perform early stopping. For example in this example, it will monitor val_loss and if it has not gone down within 10 epochs, the training will stop.
csv_logger — Logs the monitored metrics/loss to a CSV file
lr_callback — Reduces the learning rate of the optimizer by a factor of 0.1 if the val_loss does not go down within 5 epochs.
Finally, model.fit() is called with train_gen_aux as the training data, valid_gen as the validation data. The function get_steps_per_epochcomputes the number of steps given the size of the dataset and the batch size.
And then we can evaluate the model on test data using,
# Evaluate the modeltest_res = model.evaluate(test_gen_aux, steps=get_steps_per_epoch(500*50, batch_size))
If you run the training, you should get an output similar to the one below.
Epoch 1/502813/2813 [==============================] - 1465s 521ms/step - loss: 2.0031 - accuracy: 0.5557 - val_loss: 1.5206 - val_accuracy: 0.6418Epoch 2/502813/2813 [==============================] - 1456s 517ms/step - loss: 1.3250 - accuracy: 0.6777 - val_loss: 1.4021 - val_accuracy: 0.6622...Epoch 23/502813/2813 [==============================] - ETA: 0s - loss: 0.1268 - accuracy: 0.9644Epoch 00023: ReduceLROnPlateau reducing learning rate to 9.999999974752428e-08.2813/2813 [==============================] - 1456s 518ms/step - loss: 0.1268 - accuracy: 0.9644 - val_loss: 1.2681 - val_accuracy: 0.7420
and on test data,
782/782 [==============================] - 84s 107ms/step - loss: 1.0672 - accuracy: 0.7874
The model has reached around ~75% accuracy on the validation data and ~78% on the test set. This accuracy is a very respectable accuracy compared to state of the art performance (which is around 85%).
The content of this article is based on my latest upcoming book “TensorFlow 2 in Action” by Manning. Don’t be disheartened, if you don’t want to check the book out, the code for the book is available for free on Github and can be accessed here.
Here’s how this compares to some of the other attempts discussed in the book. Specifically,
inception-net-v1 data-aug + early-stop — An Inception Net v1 model (with data augmentation and early stopping)
Minception-net — A modified version of Inception Resnet v2 (trained from the scratch)
Inception-Resnet-v2 transfer-learn — Inception Resnet v2 model pretrained on ImageNet
That was a very quick hands-on guide as to how you can use Transfer Learning with TensorFlow to quickly get a model up and running. Transfer Learning enables you to transfer the knowledge of a pretrained model for a downstream task you’re interested in solving. This can be really handy in some situations. For example, if you are working on a computer vision product you might save weeks or even months by starting out with a pretrained model. So this is a very important concept to be familiar with.
In this guide we,
Learned what Transfer Learning is
Understood the dataset
Learned the nitty-gritties of InceptionNet-Resnet-v2
Downloaded a ImageNet pretrained InceptionNet-Resnet-v2
Used the downloaded model to define a new model and solve the classification task (Tiny-ImageNet) at hand quickly
If you would like to learn more about,
Data augmentation and creating data pipelines in TensorFlow,
Mechanics of convolution neural networks,
Inception net models and various design principles behind them,
Techniques like batch normalization, residual connections and,
Using Transfer Learning for image classifcation,
checkout Chapter 6 and 7 of “TensorFlow 2 in Action” by Manning (provided as an affiliate link).
Happy learning!
If you enjoy the stories I share about data science and machine learning, consider becoming a member!