Google Landmark Recognition
By: Daniel Stern
Tags: Landmark, Image Recognition, Image Classification, Transfer Learning, Data Science

MODELING

For modeling, we looked at 4 of the most popular pre-trained models for image classification. The models are as follows:

VGG-16 Inception V3 Resnet-50 EfficientNetB0

The idea was to apply transfer learning through these models to our unique image classification task. Transfer learning utilizes a network trained on a different task, which is then applied to the target task. With this methodology, we can save time training a model from scratch as well as take advantage of parameters trained on a gigantic amount of data.

How the Models Work

VGG-16

VGG-16 was one of the first breakthrough models for image recognition. This model achieved 92.7% top-5 test accuracy on the ImageNet dataset which contains over 14 million images belonging to 1000 classes. This model has 16 layers and around 138 million parameters, making it quite a bit more computationally expensive than the other 3 models. The model passes through a stack of 16 convolutional layers where small filters of size 3x3 are applied to capture patterns in the image. The model also uses techniques such as max pooling, FC layers, and a softmax layer at the end as shown in the architecture above.

Inception V3

Inception V3 is a model that was created by Google. It possesses 22 layers and around 24 million parameters. This neural network was also trained on ImageNet and utilizes a series of inception modules to derive an output. In short, these inception modules perform convolutions with several filters and max pooling then concatenates these results before sending them to the next layer.

ResNet-50

The ResNet-50 model has around 23 million parameters and 50 layers as the name suggests. The idea here is that stacking more layers gives us greater accuracy, so why not just stack more layers? The problem applying this idea was that too many layers caused vanishing gradients when updating parameters which lead to next to no parameter updating. The solution ResNet-50 developed was “shortcut connections” which reduced the vanishing gradient problem while also not adding additional parameters to the model.

EfficientNetB0

The last of the 4 models is EfficientNetB0. This pre-trained model is the latest and greatest in image recognition with around 5.3 million parameters and 18 layers. Iterations with more parameters exist from EfficientNetB1-B7, however, due to computation constraints we stuck with the simplest model. This neural network uses a technique called compound scaling which relies on the idea that adding more layers, more nodes per layer, or better resolution for filters can improve a model. Compound scaling uniformly scales all three of these dimensions with a compound coefficient which takes advantage of all 3 methods of improving a model.

Parameters and Setup

We have mentioned multiple times that we ran into computational and time constraints. As a result, we chose to benchmark the simplest versions of all of these networks. For each of the 4 networks described, we had to change the last layer of our models to a dense softmax layer of size 131 to account for our 131 unique classes. Our loss function was categorical cross-entropy, which is the most appropriate metric for multi-class image classification. We used a batch size of 240 which we felt struck the right balance between training time and training performance. Ideally, we would set our steps_per_epoch to be a factor of the length of our training data divided by our batch size (around 150 in our case), but we had to compromise with a more conservative 20 steps, meaning our model is only seeing a portion of our data. More epochs generally allows for a model to fit better, and we limited these to 10 for each model. Lastly, we did not train this size of a model on the VGG-16 network due to the larger number of parameters causing much higher training time.

RESULTS

Model Results

Best Model

EfficientNet was clearly the best performing model with a test accuracy of 16.86% on the 470 test images from 131 landmarks. While this accuracy percentage may seem low, it actually performed quite well in relation to other competition submissions. It is often the convention in multi-class classification to use top-k categorical accuracy, which marks a prediction as correct if the correct class is one of the top 3 classes predicted. When using this accuracy metric, EfficientNet achieved a 100% test accuracy with only 2 epochs and 5 steps per epoch. We wanted to see the true accuracy performance of the model so the k value was returned to 1.

CONCLUSION

Next Steps

In projects that utilize convolutional neural networks for image classification, it is important to understand both how the model is making correct predictions and identifying why it makes mistakes. A good way to do this is to implement feature extraction to see what the model determines is important within an image in making the classification. Feature extraction could be used as a tool in identifying types of images that give our model issues (i.e People in the foreground, different perspectives or if it just the distractor images in the dataset). The use of feature extraction feeds into our next step of model improvement. Parameter tuning to improve accuracy can in part be informed by the learnings from feature extraction. We could change the batch size, number of layers and the learning rate just to name a few of the possible adjustments. Also, given both time constraints and computational constraints, we did not run huge models. Utilizing Google Cloud as opposed to Colab could improve the processing efficiency of model fitting and enable the creation of neural networks with more epochs. If we find success in these domains, we will explore ensembles of various convolutional neural networks to capture the best qualities of each.

Reflections and Lessons Learned

The first thing that became evident to us was that model prep which included our data cleaning, filtering and populating our folders took a large majority of our time. Model implementation can obviously take time and be complex in its own right, but certain data cleaning issues like the broken links were very challenging for us and definitely should not be overlooked.

Another takeaway for us came from our decision to use a relatively popular dataset. Popular datasets like Google Landmark recognition obviously come with an abundance of resources and previous attempts. However, issues may arise when utilizing outside resources into our code. In summary, using resources or looking at past attempts at a project may provide helpful code or act as a problem-solving guide. There is no guarantee that it will integrate appropriately with other aspects of the code.

Lastly, we were able to find success with small toy examples looking at a limited amount of data despite some of our initial struggles with data pre-processing. Using small “testing” datasets is essential when dealing with large data sets like this Google Landmark Competition, and is not something we will neglect in future projects.

We hope this blog can be a guide for future Image Classification projects and welcome any feedback from readers. It was a great experience to work a project from start to finish with all of the obstacles along the way.

References:

Dataset

https://www.kaggle.com/google/google-landmarks-dataset

TensorFlow Tutorials

https://www.tensorflow.org/tutorials

Cat Dog Example https://www.tensorflow.org/tutorials/images/transfer_learning#feature_extraction

Top 4 Pre-Trained Models https://www.analyticsvidhya.com/blog/2020/08/top-4-pre-trained-models-for-image-classification-with-python-code/

Load Image Tutorial

https://www.tensorflow.org/tutorials/load_data/images

Top-k Metric

https://keras.io/api/metrics/accuracy_metrics/#topkcategoricalaccuracy-class

TensorFlow Image Classification

https://www.tensorflow.org/tutorials/keras/classification