# A Review of Transformer-Based Approaches for Image Captioning
## 2 Datasets and Evaluation
### Datasets  
The datasets have been used for training, validation and evaluation. The data is a crucial factor of the performance of many of the machine learning algorithms. The availability of vast amounts of data has resulted in better-trained models with significantly improved levels of accuracy. Table 1 shows a listing of the main datasets that have been used in relation to image captioning. As can be seen from the sizes, the general trend has been towards larger and larger datasets in terms of number of images and the language features (captions, questions and other textual annotations). The collection and curation of captions and other language features of the large-scale datasets tends to be automated rather than human-annotated, given the high cost of human annotation. The textual description (image-text pairs) for these large-scale datasets are largely acquired through the alt-text HTML attribute of images from the internet. Despite the fact that these large-scale datasets are noisier and less clean, they have proven to be effective in vision-language pre-training. For the smaller datasets, the annotation is commonly created through human annotators via services such as the Amazon Mechanical Turk [46]. The categories of the datasets locate inspiration to a large extent from WordNet [47; 48], together with its hierarchical structure and synsets. The synsets are sets of cognitive synonyms, each expressing a distinct concept [47]. Some of the categories that occur in many of the datasets are taken from wordnet subtrees and include _mammal, bird, fish, reptile, amphibian, vehicle, furniture, musical instrument, geological formation, tool, flower, fruit_[49]. Datasets such as MS COCO [50] tend to have a high proportion of common objects such as people, animals, furniture and vehicles. The large-scale datasets, such as OpenImages [51], ALT200M [52], WebImageText [53] and ALIGN [54] have a higher variety of categories due to their sheer scale and method of collection (automated, from web images). WebLI [55] ramps up the scaling further and is made up of 10 billion images and the corresponding textual descriptions. Three datasets worthy of particular mention are ImageNet [49; 56], MS COCO [50] and Nocaps [57]. ImageNet spurred a lot of research in deep learning models for computer vision and it contributed significantly to better models related to vision tasks such as image classification and object detection. ImageNet was the basis for the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [7], which saw convolutional neural networks(CNN) come of age and achieve and surpass human-level performance in some of the vision tasks. MS COCO [50] has also been a major driver in the development of computer vision applications. It has fostered state-of-the-art algorithm development in areas such as image classification, object detection, image segmentation and image captioning. Many of the image captioning algorithms developed since 2014 have used this dataset as a basis for evaluation and validation. In developing their method for deep visual-semantic alignments for generating image captions, [PERSON] and [PERSON] [27] used a subset of the MS COCO Captions 2014 dataset and divided the subset into training, validation and evaluation subset. This division has come to be called the _Karpathy split_ and has been used by many subsequent models as a basis for offline comparison for different algorithms. In the [PERSON] split, the validation set of the MS COCO 2014 dataset is divided into a 'val' split and a 'test' split, each with 5000 images, and a'restval' split with 30,504 images. The 'train' split has 82,783 images. The'restval' and 'train' splits can be combined to make a total of 113,287 training images. The online evaluation server associated with the MS COCO dataset has also been useful for algorithm development. The server is hosted by CodaLab [68] and has served as a tool for different state-of-the-art models to be objectively evaluated and compared against each other.