Image Captioning with Attention: Part 2
By: Artyom Makarov
Tags: Deep Learning, Image Captioning, Computer Vision, Pytorch

Model Training

In the first part of the article, we have covered the overall architecture of the Encoder-Decoder model for image captioning.

Now let’s discuss the training process in detail. You can find the training notebook via the GitHub link.

Hyperparameters

Note the smaller batch_size results in a stronger regularization effect and makes it easier to fit one batch in memory. I’ve chosen a batch size equal to 64 .

results in a stronger regularization effect and makes it easier to fit one batch in memory. I’ve chosen a batch size equal to . vocab_threshold - the total number of times that a word must appear in the captions before it is used as part of the vocabulary. The higher the threshold is, the stricter limit we impose on creating our vocabulary.

- the total number of times that a word must appear in the captions before it is used as part of the vocabulary. The higher the threshold is, the stricter limit we impose on creating our vocabulary. Based on the experimentation the number of epochs was set to 14 . In practice, we keep training as long as the training and validation errors keep dropping.

Transformers

Data transformation

First, we resize the original image, performing transforms.Rezize(256) and randomly crop to get a 224x224 image sample- transforms.RandomCrop(224) .

and randomly crop to get a 224x224 image sample- . Subsequently, we flip the sample horizontally transforms.RandomHorizontalFlip() , convert to a tensor transform.ToTensor() and normalize.

, convert to a tensor and normalize. Note that normalization is applied to all channels (depth=3) of an image sample - transform.Normalize((0.485, 0.485, 0.485), (0.229, 0.229, 0.229)) , given the mean of 0.485 and a standard deviation of 0.229 .

Training loop

To complete the training on a single epoch, we define the function which receives the following arguments:

epoch — number of the current epoch.

encoder — model’s Encoder, which is set to evaluation mode eval() .

. decoder — model’s Decoder, which we aim to train.

optimizer — model’s optimizer (Adam in our case). Adam is a common choice for the training, that possesses the properties of both AdaGrad and RMSProp ⁶.

criterion — loss function to optimize. We use Cross-Entropy Loss CrossEntropyLoss() that comprises an effect of Negative Log-likelihood NLLLoss() , applied to probabilities, produced by softmax function.

that comprises an effect of Negative Log-likelihood , applied to probabilities, produced by softmax function. num_epochs — total number of epochs. We used 14 epochs to train the model.

epochs to train the model. data_loader — specified data loader (for training, validation, or testing).

write_file — file to write the training logs. We store the stats in two separate txt files.

files. save_every - to save the results after each trained epoch.

Note, we store the captions that we train on without the first word in the captions_train variable and target captions without the last word in captions_target variable.

The full code for the training loop is shown below.

Training function

Validation loop

The validation function follows the same logic as the training one, except for the BLUE score we calculate for the model’s hypothesis.

At each validation step:

We pass the output indices terms_idx , generated by the model to the get_hypothesis() function and get the list of hypothesis hyp_list for image batch.

, generated by the model to the function and get the list of hypothesis for image batch. Next, we populate the hypothesis and references lists with hyp_list and caps_processed correspondingly. The first list stores all hypotheses we get within a single epoch. The second list contains all processed target captions caps_processed , returned using get_batch_caps() function.

and lists with and correspondingly. The first list stores all hypotheses we get within a single epoch. The second list contains all processed target captions , returned using function. Calculate the BLEU score using the corpus_bleu() function from the NLTK package.

BLEU score in a nutshell

BLEU score is a standard metric, which is widely used in NLP and CV domains for evaluating the machine-generated translation or caption against the human one. The details of the BLEU algorithm can be found in the original paper¹ and this famous Deep learning course by Andrew Ng.

I also provide the formula and supportive example to demonstrate the BLEU score calculation for a single image, according to the NLTK implementation.

BLEU score calculation

BP — “brevity penalty”. We set it to 1.0 if the length of a candidate is the same as any translation length.

Brevity penalty calculation

Let’s have a look at BLEU scores estimation based on 1,2,3 and 4-gram precision for a single image from the batch.