# A Review of Transformer-Based Approaches for Image Captioning
## 6 Discussion and Analysis
### Datasets and Metrics  
Table 5 shows the datasets and the metrics used by the non-VLP and VLP models. With regard to the datasets, it is seen that the MS COCO dataset has been used by all the models. Results reported based on the MS COCO dataset therefore offer a good basis of comparison of different models. The Visual Genome dataset is largely used for pre-training of the feature extractor, especially in cases where the attributes and object relationships are required to provide richer features.  
The methods that use VLP usually require much larger datasets for the pre-training. Therefore, the some of the VLP methods resulted in the creation of large-scale datasets. Illustrative examples are the ALT200M [52], WebImageText those used by CLIP [53] and ALIGN [54] datasets. CoCa uses the JFT-3B dataset [114]. CC3M and CC12M are variants of Conceptual Captions; WIT is the Wikipedia-based image text dataset. Together with large datasets, the VLP methods entail increasingly larger model sizes.  
The metrics used have been standard, which facilitates the comparison of the various models. The metrics are useful for offline as well as online evaluation on the official MS COCO test server. The COS-Net model [91] makes use of the CHAIR metric [115], which is used to measure object hallucination. However, since this is not used by the other models, it cannot be used here for comparison purposes.  
\begin{table}
\begin{tabular}{p{113.8 pt} p{284.5 pt}} \hline \hline
**Model** & **Major Contributions** \\ \hline OSCAR [38] & The use of object tags as anchor points for cross-modal representation learning; a novel pre-training objective. \\ \hline Unified VLP [109] & A single unified multilayer transformer network for encoding and decoding that is universally applicable to multiple downstream tasks. \\ \hline VinVL [39] & An enhanced object detection model; they demonstrate that visual features matter significantly in VL models. \\ \hline VIVO [40] & Creation of visual vocabulary by pre-training on image-tag pairs rather than image-caption pairs. \\ \hline LEMON [52] & An empirical study of the effects of scaling up VLP; ALT200M, a large-scale pre-training dataset based on alt attribute of internet images. \\ \hline SIMVLM [103] & A single Prefix Language Modeling objective which enables bi-directional contextualized representation and autoregressive generation. \\ \hline CLIP [53] & An efficient and scalable way to learn image representations using a large dataset of image-text pairs collected from the internet. \\ \hline ALIGN [54] & A large-scale dataset of 1.8 B image-text pairs from the internet without expensive processing; useful for image representations via pre-training. \\ \hline OFA [106] & A “one-for-all” seq-to-seq framework that is task-agnostic, modality-agnostic and task comprehensive. \\ \hline CoCa [110] & A Contrastive Captioner that pre-trains a foundation model jointly with contrastive and captioning losses combined into a single objective. \\ \hline GIT [111] & A simplified architecture of an image encoder and a text decoder under one language modeling task. A new scheme of generation-based image classification. \\ \hline Universal Captioner [112] & A model capable of generating in/out-of-domain captions characterized by the natural descriptive style and fluency of human captions. \\ \hline PalLI [55] & Joint scaling of vision and language components resulting in a model that performs vision-language tasks in many languages; WebLI, a large-scale database with 10 B image-text pairs in more than 100 languages. \\ \hline BLIP-2 [113] & A lightweight querying transformer that acts as a bridge between two off-the-shelf large-scale unimodal transformers pre-trained and then frozen; results in compute-efficient vision-language pre-training. \\ \hline \hline \end{tabular}
\end{table}
Table 4: Major contributions of the VLP transformer-based models.