VIVO: Now AI😏 Can Write Better Captions Than Most Humans
By: Parth Chokhra
Tags: Machine Learning, Artificial Intelligence, Deep Learning, Data Science, Data

First, in the pre-training stage, authors train a Transformer based model on image-tag pairs for tag prediction where it learns a joint representation of visual and text input.

In fine-tuning authors train the model on a smaller dataset of image-caption pairs to learn how to generate captions based on the given images and tags. Below is an example of how pretraining is done.

Input A: “[A] holding [B] …”.

Input B: The corresponding object tags detected. In this case, it is a “person” and “dog”

Output: “a person holding a dog”

While the caption templates are learned from image-caption pairs, the object tags to be filled may refer to novel visual objects that are unseen in image-caption pairs (but seen in image-tag data in this example).

Training the model in this fashion allows zero-shot generalization to novel objects for image captioning.

Model Architecture

The model consists of multiple Transformers layers to encode the input into a feature vector and linear layer with softmax to generate text captions for the visual object in the image.

Overview of our VIVO pre-trained Transformer model. Courtesy: VIVO

Implementation Detail

In pre-training, the Transformer model is trained on large scale dataset. Unlike most Vision-and-language(VLP) pretraining methods which rely on image-caption pairs, VIVO pretraining is conducted solely on image-tag pairs. Since we do only need image tag pairs it often easy to gather large dataset or annotate large dataset. The training objective is to predict the missing (masked) tags given a bag of image-level tags and image regions.

After pre-training, the downstream task was to finetuned Transformer model on a dataset where both captions and tags are available. The tags can also be automatically generated using a pre-trained tagging or detection model. Given image regions and tags, the model learns to predict the conditional caption sentence where some positions are randomly masked out.

Authors used object detector from UpDown to extract image features which are then concatenated to dimensions of the bounding boxes to form a 2054-dimension vector (2048D for the visual features and 6D for the bounding box encoding). This detector is trained on Open Image Dataset to detect all object with tags.

Experiments and Results

Novel Object Captioning: The model is compared to Updown and Oscar which previously were state-of-the-art results on nocaps benchmark. The training data for baselines is the COCO mark. Using VIVO pretraining on Open Image Dataset the model has achieved significant improvement compared to all prior works. Here is the table showing the performance compared to other models.

Although object tags are used in both VIVO pre-training and fine-tuning stages, we show that the model’s capability of generating captions that precisely describe novel objects at inference time attributes largely to pre-training

Evaluation of nocaps validation and test sets. Courtesy: VIVO

Visual-Text Alignment

To further explore the effects of pretraining in learning visual vocabulary which aligns image regions with the object tags the authors performed a simple test.

They extracted image region using the UpDown object Detector and generate captions s from the captioning model with VIVO pre-training.

After identifying the novel objects in the generated captions, they feed the novel object tags, together with the extracted image region features, to the VIVO pre-trained Transformer model. The output of the last encoder layer is used as the contextualized representation of the corresponding input.

Then cosine similarity is calculated between the representation of each pair of image region and object tag and results show they precisely match.

Conclusions

The authors of the paper presented a weakly supervised learning approach of training image captioning model in i.e. is first training a Transformer based model for image-tag pairs and then finetuning it with image-captions pair.

The experiments on the nocaps benchmarks show that the model achieves a zero-shot generalization to novel object for image captioning. Due to this, the best model trained on VIVO pretraining method achieves state-of-the-art accuracy that surpasses the human CIDEr score on nocaps. A detailed analysis reveals that the generalization is attributed to a large degree to the visual vocabulary learned in model pre-training.

References