Understanding Automatic Photo Caption Generation Using Keras
By: Dhanraj Bhedi
Tags: Photo Captioning, Computer Vision, Naturallanguageprocessing, Image Captioning, Artificial Intelligence

In the past few years, Automatic photo caption generator has attracted the attention of many researchers of artificial intelligence. From driverless cars to speech recognition, Artificial intelligence is making everything possible. It has become a hot topic in Industry as well as academia. Automatic photo caption generation requires computer vision (CV) to understand the image and natural language processing (NLP) to generates descriptions relevant to images. CV and NLP are subfields of artificial intelligence that continuously changing the world. The application of image caption is extensive and significant, for example, human-computer interaction.

Introduction

If you see the image and you easily can think of what is in the image but how a machine can tell what the image is describing?

Can you describe the below image?

Example

One person will say “Craws standing on the ground in the field” and the second will say “two birds near one another in the field”. This is easy for human beings to describe but for machines. Let’s see how a machine can describe an image.

Motivation

It will be a very useful product for blinds which can describe the surroundings.

It will be very useful in autonomous vehicles by providing more accurate information of scenarios to boost up decision making.

Also useful in surveillance system by giving alarm with the description of suspicious activity.

Dataset

There are many open-source datasets available for Image captioning, like Flickr 8k (containing8k images), Flickr 30k (containing 30k images), MS COCO (containing 180k images), etc. But for the purpose of this case study, I have used the MS COCO dataset.

Here I have used only 30k Random images of the MS COCO dataset.

Data Pre-processing for Captions

Here I have used some pre-processing tasks like removing punctuations, lowering case, tokenizing, remove one letter word like ‘a’.

Code for processing captions

Here I have considered only those words that have occurred more than 10 times.

Reconsidering Captions

Now I have created a dictionary for conversion between a word to index and index to a word.

For Index to word and Word to Index

Data Pre-processing For Images

I have used transfer learning on InceptionV3 model which is trained on 1000 classes and more than million images.

So, I convert image to that dimension which is required for inceptionV3 model then extract the feature from the Image.

Processing for Images

Methodology

A captioning model depends on two main components, a CNN and an RNN.CNN (Convolutional Neural Network) is used for feature extraction. RNN (Recurrent Neural Network) used for generating sequence data such as generating a sequence of words.

Here we are using 2 different models, one which built on LSTM (Long Short Term Memory) and the second which built on GRU (Gated Recurrent Unit).

Flow of Model

Word Embedding

Here I have to convert the caption into vectors for the network. So, I have used the pre-trained GLOVE embedding of billion words to map every word (index) to a 200-long vector.

Word Embedding

Now, for all unique words in our vocabulary, we create an embedding matrix which will be loaded into the model before training.

Creating a matrix for words

Model Architecture

Model Definition

Model Architecture

Here I am stop training for the embedding layer because we have pre-trained glove embedding.

Freezing Layer

Model summary

Model Summary

Evaluation

Our model will generate only one word at a time. So, I am iterating the operation until I got the end of sequence or max length. By this iteration, we got a sentence that describes the image.

Function for generating a sentence

Result

Output 1

Output 2

Conclusion and future work

Here I have used the LSTM and GRU-based model and I can conclude that if you want faster training and you have less data then GRU gives more accurate results otherwise LSTM gives the best results.

Few points to improve this solution:

Use a large dataset to improve the result

Try to tune the hyperparameter of the model

Use beam search approach instead of simple iterations over the model

Use the attention layers to improve the model.

References

https://www.analyticsvidhya.com/blog/2018/04/solving-an-image-captioning-task-using-deep-learning/

https://towardsdatascience.com/image-captioning-in-deep-learning-9cd23fb4d8d2

https://www.tensorflow.org/tutorials/text/image_captioning

https://software.intel.com/content/www/us/en/develop/articles/inception-v3-deep-convolutional-architecture-for-classifying-acute-myeloidlymphoblastic.html