Medical Image Captioning on Chest X-Rays
By: Ashish Thomas Chempolil
Tags: Data Science, TensorFlow, Medical, Image Captioning, Deep Learning

5. Existing Solutions

Image Captioning with Keras | by Harshall Lamba: Here he has used flicker 8k images as the dataset. For each image there are 5 captions and he has stored them in a dictionary. For data cleaning, he has applied lowercase to all words and removed special tokens and eliminated words with numbers (like ‘hey199’, etc.). After extracting unique words he removed all those words whose frequency was less than 10 in the entire corpus so that model can be robust to outliers. Now he added “startseq” and “endseq” to each captions. For preprocessing of the images he has used inception v3 model with Image net dataset weights. Images were sent to this model and the output of the second last layer was taken (2048 size vector, bottleneck features) and saved these vectors. For preprocessing of captions he has tokenized the captions and also found out the maximum length of the sentence in the captions (found to be 34) so that he can use it for padding. He has used pretrained Glove vectors for word embeddings (200 dimensions) and also set trainable = False for the embedding layer.

This is the model architecture he has used for training which is self explanatory. The output from the above model gives a probability value for which word has the highest probability to come next in the sentence. Here he is using greedy search method (gets the word which has the highest probability to come next in the sentence). The model was then trained for 30 epochs with the initial learning rate of 0.001 and 3 pictures per batch (batch size). However after 20 epochs, the learning rate was reduced to 0.0001. Loss used is categorical cross entropy while optimizer used was Adam.

Image Captioning using Attention Mechanism | by Subham Sarkar | The Startup: Here he has used attention mechanism for image captioning. The dataset used is flickr8k same as before. Here the train size was 6000 images, validation data size was 1000 images and test data size was 1000. For preprocessing he has removed punctuation, numeric values and single characters. Then he creates a dataframe with columns filename and captions. He has reshaped each image into 224*224*3 before feeding it to a pretrained on ImageNet VGG-16 model. Here he has only taken convolutional part (include_top=False). He has removed the last dense layer and has taken the output from the second last layer as backbone features.

For captions, he has tokenized and added <unk> as the token for recognizing those words which are not in the vocabulary. He has padded the tokenized sequence with that of the maximum length of sentence found in the corpus (=39).

For attention mechanism he has used both global attention and local attention and did not combine them but instead gave an option of choosing which attention to use, here in the code file he has used only local attention. Global attention looks at the image as a whole while local attention focus on specific parts of an image (in the context of image captioning). This is done by considering the hidden states to derive the context vector. The global attention considers all the hidden vectors while local attention considers a specific window of hidden vectors per target word.

The encoder here is the VGG-16 model. The decoder is RNN. Here he is using GRU. He has used teacher forcing method here whereby he input the actual tokens as the next input instead of the predicted tokens for the decoder gru at different time steps during training. The different steps for global attention and local attention are explained below:

Attention Mechanism:

The features that are considered as encoder hidden states are features from VGG-16 here. The previous decoder hidden state and decoder output is passed through decoder gru to generate a new hidden state and prediction probability score. Using all the hidden states from encoder and new decoder hidden state we will calculate the attention scores. These attention scores are then softmaxed to give probability scores (which all part should be given attention to) Using this probability scores he calculated the context vector by multiplying each scores with encoder hidden states (here cnn features) then gives it to GRU. This is repeated until it reaches the padded length.

Now for evaluating the captions he has used two methods beam search and greedy search. Here the number of predictions considered for beam search was b=3,7,10 (top b predictions). Beam prediction gets the caption with the most probability (i.e. here he sums up the probability values for each word and evaluates the final probability of that sentence/caption).

Here he has used the BLEU score as the metric to evaluate the prediction. He built a custom loss function which only penalizes based on the word that are in the caption. He has used an Adam optimizer.

A good blog for further understanding of Attention mechanism.