Package: nic
neural-image-captionA simple Python API built on top ofTensorFlowfor neural image captioning withMSCOCOdata.Table of contentsDescriptionInstallationMSCOCO APINIC ModelTraining on Google ColabReferencesDescriptionThenicAPI has two main purposes:working with the MSCOCO datasetThe data can be downloaded, preprocessed and then loaded into Python objects as expected by TensorFlow.training a neural network model for image captioningA deep neural network model with sequence-to-sequence architecture can be easily defined, trained on the dataset and then used to caption images.These are discussed in more detail in the following sections.InstallationThe API is available on PYPI and can be istalled with pip:pip install nicMSCOCO APIThe MSCOCO dataset consists of more than 100 000 captioned images. Each image is "paired" with a few descriptions (in English) of what can be seen on it.ThenicAPI makes it possible to download the dataset, preprocess the data and load it into Python objects used to train neural networks. We'll look into each of these next.Note that the dataset is very big so downloading and preprocessing it will take up a lot of space. At the time of writing this,an archive fileof the dataset is between 10 and 20 GB. This is why getting rid of the original data might be a good idea once it is preprocessed.First we need to import the API (and TensorFlow).importtensorflowastfimportnicDownloadingThen we can download the dataset (fromhere).mscoco_dir=r"mscoco"version="2017"nic.dp.download_mscoco(mscoco_dir,version)The dataset has train and validation splits so we will create a test split from the train data. Usually 20% of the samples are used for testing:nic.dp.split_out_test_data(mscoco_dir,split=0.2,version=version,verbose=True)A note for those who may want to use the original MSCOCO data for something else too:The train images (randomly) selected for testing are moved frommscoco/train2017to a separate directory namedmscoco/test2017. Their annotations are extracted fromannotations/captions_train2017.jsontoannotations/captions_test2017.jsonbut this extraction simply removes the annotations from the 'annotations' list in the first file and creates the second file which only contains the extracted annotations like so:{"annotations": <annotations>}.A copy of the original train captions file is created as back up so the original structure of the dataset can be restored by moving the images back totrain2017, deleting thecaptions_test2017.jsonfile and restoring the backup file with train captions.PreprocessingNext, we preprocess the dataset by calling thepreprocess_datafunction. We provide this function with the path of the MSCOCO directory, the path where to store the preprocessed data, the meta tokens to be used when preprocessing captions, the maximum number of words (if needed) to include in the dictionary extracted from the captions, and some image options.The image options describe the way in which images are preprocessed. Image preprocessing involves 'preparing' images for a specific CNN encoder and optionally extracting features for images by running them through the encoder. The second part is useful when doing transfer learning with the CNN encoder module of the model being frozen. Extracting the features once and reusing them to train the other model layers is much more efficient than the alternative.The image options are as follows:model_nameThe name of the CNN encoder to preprocess the images for. This model is looked up intf.keras.applicationsand itspreprocess_inputmethod is called on batches of imagestarget_sizeThe spatial size of the image, as expected by the chosen CNN encoderfeature_extractorA callable taking and returning atf.Tensor. If provided, it will extract features for batches of preprocessed imagesbatch_sizeThe batch size to use when preprocessing (and extracting features for) the imagesAs we will see in a moment, the API provides a function that loads preprocessed data into atf.data.Dataset. For those interested, here is how the preprocessed data looks on disk:the data is stored in a directoryDwhich has three subdirectories -train,testandvaleach of the subdirectories has a subdirectory namedimageswhich stores preprocessed images and optionally a subdirectory namedfeatureswhich stores features extracted for the images. Preprocessed images and image features are pickledtf.Tensors, the file names are simply<image_id>.pcleach of the subdirectories also contains a file namedcaptions.pcl. It contains a pickled dictionary mapping image ids (int) to a list of str captions (the original captions enclosed with the start and end meta tokens)thetrainsubdirectory has another file -tokenizer.json. This is the JSON representation of atf.keras.preprocessing.text.Tokenizercreated from the train captionsIn this example we will preprocess the data forInception ResNet v2.data_dir="data"encoder=tf.keras.applications.inception_resnet_v2.InceptionResNetV2(include_top=False,weights="imagenet",pooling="max")encoder=tf.keras.Model(encoder.input,encoder.layers[-1].output,name="inception-resnet-v2")image_options=nic.dp.ImageOptions(model_name="inception_resnet_v2",target_size=(299,299),feature_extractor=encoder,batch_size=16)meta_tokens=nic.dp.MetaTokens(start="<start>",end="<end>",unknown="<unk>",padding="<pad>",)max_words=Nonenic.dp.preprocess_data(source_dir=mscoco_dir,target_dir=data_dir,version=version,image_options=image_options,meta_tokens=meta_tokens,max_words=max_words,verbose=True)Loading preprocessed dataPreprocessed data can be loaded with theload_datafunction. It takes the path of the directory where preprocessed data is stored, the type of data to load ('train','val'or'test') and a boolean value indicating whether to load features or preprocessed images:train_data=nic.dp.load_data(data_dir,type="train",load_as_features=True)test_data=nic.dp.load_data(data_dir,type="test",load_as_features=False)The data is loaded into atf.data.Datasetwhich yields 3-tuples whose components aretf.Tensors:the 3D image tensor or features vector (ifload_as_featuresis set toTrue)an integer vector which represents a caption for the image, without the end meta token at the endan integer vector which represents the same caption but this time without the start meta token in frontThe shape of the caption vectors is(max_caption_length,)and shorter captions are post-padded with 0 (the index of the padding meta token). The shape of the image or features tensor depends on the chosen CNN encoder.Keras models are typically trained withtf.data.Datasetobjects which yield elements with a different structure (not  3-tuples). In order to train a model with the datasets returned byload_data, we'd need to customisefit, as explainedhere.As describedlater, thenicAPI can also be used to define and train a model with the MSCOCO dataset. TheCustomModelclass defined withinniccan be used as an example for customizingfitto work with the datasets returned byload_data.There are a few more API functions that work with preprocessed data. Thetokenizercan be loaded like this:tokenizer=nic.dp.load_tokenizer(data_dir)Captions can be loaded into a dictionary mapping integers (image ids) to lists of strings (the original captions enclosed with the start and end meta tokens):val_captions=nic.dp.load_captions(data_dir,type="val")Images (preprocessed for the chosen CNN encoder) or their corresponding features can be loaded into atf.data.Datasetwhich yields pairs of the images/features and the image id:test_images,count=nic.dp.load_images(data_dir,type="test",load_as_features=False)Vocabulary and features sizes can also be obtained:vocabulary_size=nic.dp.vocabulary_size(data_dir)features_size=nic.dp.features_size(data_dir)NIC ModelThe other main part of thenicAPI is a neural network model that can be easily defined, trained on the MSCOCO dataset and then used to caption images.The model has a Seq2Seq architecture which is depicted below.Images are represented as 3D tensors which are fed into a CNN. The resulting feature vectors are transformed and fed into an RNN, as the initial hidden state vectors.Captions are tokenized and each token is represented as a vector from a word embedding. The word embeddings are fed into the RNN as inputs.The hidden state vectors at each time point (caption length) are transformed and projected over the vocabulary words/terms.During training, the word projections are used to calculate the loss (categorical cross entropy). During inference, the projections are used to generate a word distribution that is used to select the next word in the caption.The CNN image encoder is typically a pretrained model, like Inception ResNet v2. The rest of the model, visualised withtf.keras.utils.plot_model, looks like this (the RNN's hidden size is 512 in this case):The model is largely similar to the one describedhere.Defining the modelFirst we need to import the API:importnicThe CNN encoder module can be any model (built with TensorFlow 2) that transforms an image (3D tensor) into a vector. Remember that the encoder is important when preprocessing data too, as mentioned in theMSCOCOsection. ThenicAPI makes it easy to use Inception ResNet v2 via the following function call:pooling="max"encoder=nic.define_encoder_model(pooling)This returns theInception ResNet v2model trained on ImageNet with the top layer removed and global pooling applied to the last convolutional layer so that the output is a vector.poolingcan be"max"or"avg".The rest of the model (the RNN, word embeddings and so on) is referred to as 'decoder' below for simplicity (even though that is not what is typically called a decoder).The decoder can be defined with thedefine_decoder_modelfunction. It needs to be passed the features size, vocabulary size, embedding size and some options for the RNN module. The first two can be obtained via the API from preprocessed data; docs are available for theRNNOptions(as well as for every public object fromnic), usehelp(nic.RNNOptions)in an interpreter. The embedding size defaults to the RNN's hidden size.data_dir=r"data"rnn_options=nic.RNNOptions(size=256)embedding_size=Nonedecoder=nic.define_decoder_model(nic.dp.features_size(data_dir),nic.dp.vocabulary_size(data_dir),rnn_options,embedding_size,name="nic-decoder")The two modules can be connected into a single model:model=nic.connect(decoder,image_shape=(299,299,3),encoder=encoder,name="nic")Theimage_shapeargument must be a three-tuple of integers - the shape of the input images, as expected by the encoder.If the encoder module is going to be the default one - Inception ResNet v2, the model can be defined like so:model=nic.define_model(nic.dp.vocabulary_size(data_dir),rnn_options,embedding_size,pooling)Training the modelThe model, or the decoder module only, can be trained on preprocessed MSCOCO data. First, the model (or decoder) needs to be compiled:compiled_model=nic.compile_model(model,learning_rate=0.0001)A compiled model can be trained withtrain_model:checkpoint_dir=r"training/checkpoints"tensor_board_dir=r"training/tensor_board"history,test_metrics=nic.train_model(model=compiled_model,path_to_data=data_dir,is_decoder_only=False,batch_size=32,buffer_size=1024,tensor_board_dir=tensor_board_dir,tensor_board_update_freq="epoch",checkpoint_dir=checkpoint_dir,checkpoint_freq="epoch",learning_rate_decay=0.9,decay_patience=3,perplexity_delta=0.001,min_learning_rate=0.,early_stop_patience=3,max_epochs=10,shuffle_for_each_epoch=False,initial_epoch=0)This function trains the compiled model for at mostmax_epochsepochs, possibly shuffling the train data prior to each epoch (shuffle_for_each_epoch).  Resuming a training process is as easy as settinginitial_epochto the number of the last completed epoch andincreasingmax_epochs.The initial learning rate is the compiled model's learning rate, if the process is started from scratch; restored models (mention in a bit) come with their optimizers which include the latest learning rate. If the validation perplexity does not improve with at leastperplexity_deltafordecay_patienceepochs in a row, the learning rate is reduced my multiplying it withlearning_rate_decay($lr = decay * lr$​​). Ifearly_stop_patiencelearning rate changes still lead to no perplexity improvement (or the loss becomes NaN), the training process is terminated.TensorBoard logs go totensor_board_dirwithtensor_board_update_freqfrequency. Checkpoints (SavedModels) go tocheckpoint_dirwithcheckpoint_freqfrequency.buffer_sizeis the size of the buffer used to shuffle the train data before training is started.More details are available in the function's docs (help(nic.train_model)).A model checkpoint can be restored like so:compiled_model=nic.restore_model(checkpoint_dir,restore_best=False)Settingrestore_besttoTruewould restore the model with the best validation perplexity. Otherwise, the latest checkpoint is loaded.Evaluating the modelA compiled model (which can also be the decoder module only) can be evaluated by computing itsBLEU-4score:meta_tokens=nic.dp.MetaTokens()bleu_score=nic.bleu_score_of(compiled_model,is_decoder_only=False,path_to_data=data_dir,batch_size=32,data_type="test",meta_tokens=meta_tokens,caption_limit=100,verbose=True)data_typecan also be"val"or even"train".meta_tokensshould be the meta tokens used when preprocessing data, which are typically the default ones so this can be omitted.caption_limitis a limit for the captions generated fromdata_typeimages. Omitting it means that there is no limit which is not a good idea for models that have not been trained for much time (as the captions are still pretty random and can be very long).Generating captionsCaptions can be generated by usingnic.CaptionGenerator. It can be created from the entire model or the decoder module only.Creating it from the decoder is restrictive as this means it will need to be fed image features, as returned by the encoder. This is useful when training the decoder as we can evaluate it without needing the images (which take up a lot of space).Creating it from the entire model is useful when evaluating the model or at inference time, as we will need to process images (not image features). Here's an example:image_options=nic.dp.ImageOptions()generator=nic.CaptionGenerator(compiled_model,meta_tokens=meta_tokens,tokenizer=nic.dp.load_tokenizer(data_dir),is_decoder_only=False,image_options=image_options)Again,image_optionsshould be the image options used when preprocessing the data (same formeta_tokens).Anic.CaptionGeneratorinstance generates captions for batches of images. A batch is represented as atf.Tensorof:image paths or 3D image tensors, when the instance is created from the entire modelfeatures vectors, when the instance is created from the decoder module onlyFor example, we can call the above generator on a batch of image paths like this:importtensorflowastfimage_paths=["images/cat.jpg","images/car.jpg",]captions=generator(tf.constant(image_paths),limit=None)A list of captions (lists ofstrtokens) is returned. To limit the length of the captions, we can setlimit.The following example shows a batch of 3D image tensors being passed to the generator:images,count=nic.dp.load_images(data_dir,type="val")images=images.batch(10)images_batch,ids_batch=next(iter(images))captions=generator(images_batch,limit=100)Similarly, we could create the generator from the decoder module:generator=nic.CaptionGenerator(decoder,meta_tokens=meta_tokens,tokenizer=nic.dp.load_tokenizer(data_dir),is_decoder_only=True)There's no need for image options as this generator will be working with image features only:images,count=nic.dp.load_images(data_dir,type="val",load_as_features=True)images=images.batch(10)images_batch,ids_batch=next(iter(images))captions=generator(images_batch,limit=100)Once we have an iterable of batches, we can create a Python generator that calls the caption generator on each batch:forcaptions_batchinnic.generate_captions(images,generator,limit=100):passThere's also a convenience function that creates the caption generator and then returns a Python generator that calls the caption generator on each batch of tensors in an iterable:batches_of_captions=list(nic.generate_captions_from_tensors(images,decoder,meta_tokens=meta_tokens,tokenizer=nic.dp.load_tokenizer(data_dir),is_decoder_only=True,caption_limit=100,))Finally, there's a high-level function that generates captions when given image paths. We also need to give it the entire model, the path to the preprocessed data, meta tokens and image options (if they are not the default ones), as well as the batch size to use and a caption limit:images_paths=["images/cat.jpg","images/car.jpg","images/nature.jpg",]captions=nic.generate_captions_from_paths(images_paths,compiled_model,path_to_data=data_dir,batch_size=32,meta_tokens=meta_tokens,image_options=image_options,caption_limit=100)image_captions=dict(zip(image_paths,captions))The returned value is a Python generator which yieldsstrs - the captions generated for the given images, in the same order. In the example above we pair each of the paths with the corresponding caption and create a mapping from the pairs.Training on Google ColabGoogle Colaboffers a Python environment with preinstalled packages like TensorFlow. It is also possible to request a GPU for a user allocated runtime. The runtimes have limited resources and even though Google Drive can be mounted, it most definitely wouldn't fit the entire MSCOCO dataset (the images in particular).To take advantage of Colab, we can:preprocess the dataset on our machines oncecreate an archive file containing image featuresupload it to Google Driveextract the features into the runtimetrain and evaluate a model using a GPUIn fact, thedatadirectory contains preprocessed MSCOCO data with image features extracted with Inception ResNet v2. Each of the two archives -max.zipandavg.zip- is a separate instance of preprocessed data; the name indicates the global pooling applied to the output of the last convolutional block when extracting image features. Theneural_image_captionnotebook can be used with any of the archive files to train and evaluate the decoder module of a model on Google Colab.ReferencesO. Vinyals, A. Toshev, S. Bengio and D. Erhan, "Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 4, pp. 652-663, 1 April 2017, doi: 10.1109/TPAMI.2016.2587640.