# A Review of Transformer-Based Approaches for Image Captioning
## 6 Discussion and Analysis
### Training and Model Parameters
\begin{table}
\begin{tabular}{l l l l} \hline \hline \multicolumn{1}{c}{**Model**} & \multicolumn{1}{c}{**VLP**} & \multicolumn{1}{c}{**Datasets**} & \multicolumn{1}{c}{**Metrics**} \\ \hline Conceptual Captions transformer [59] & ✗ & MS COCO, CC & B, M, R, C \\ Stacked attention transformer [77] & ✗ & MS COCO & B, R, C \\ Object relation transformer [79] & ✗ & MS COCO & B, S, M, R, C \\ Attention on Attention [84] & ✗ & MS COCO; ImageNet, Visual Genome & B, S, M, R, C \\ Entangled Transformer [85] & ✗ & MS COCO & B, S, M, R, C \\ Meshed-Memory Transformer [86] & ✗ & MS COCO, Nocaps, Visual Genome & B, S, M, R, C \\ X-Linear Attention Networks [87] & ✗ & MS COCO, ImageNet, Visual Genome & B, S, M, R, C \\ Image Transformer [90] & ✗ & MS COCO, ImageNet, Visual Genome & B, S, M, R, C \\ COS-Net [91] & ✗ & MS COCO & B, S, M, R, C, CH \\ OSCAR [38] & ✓� & MS COCO, CC, SBU, Flickr30k, VQA, GQA, Nocaps & B, S, M, C \\ Unified VLP [109] & ✓� & MS COCO, CC, Flickr30k, VQA v2.0; VG & B, S, M, C \\ VinVL [39] & ✓� & MS COCO, O1 v5, Objects365, VG; CC, SBU, Flickr30k; VQA, QQA & B, S, M, C \\ VIVO [40] & ✓� & MS COCO, Nocaps; QI & B, S, M, C \\ LEMON [52] & ✓� & MS COCO, Nocaps, CC, ALT200M & B, S, M, C \\ SIMVLM [103] & ✓� & ALIGN, C4, MS COCO, Nocaps & B, S, M, C \\ OFA [106] & ✓� & MS COCO, CC, SBU, VG & B, S, M, C \\ C\(\mathrm{\SIUnitSymbolMicro\SIUnitSymbolMicro\SIUnitSymbolMicro}\) & ✓� & MS COCO, Nocaps, JFT-3B, ALIGN & B, S, M, C \\ GIT [111] & ✓� & MS COCO, CC, SBU, VG, ALT200M & B, S, M, C \\ Universal Captioner [112] & ✓� & MS COCO, Flicker30k, SBU, CC, WIT, YFCC100M, OI & B, S, M, R, C \\ PalLI [55] & ✓� & WebLI, MS COCO, Nocaps, TextCaps, VQA & C \\ BLIP-2 [113] & ✓� & MS COCO, VG, CCM, CC12M, SBU & B, C \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Datasets and metrics used by VLP and non-VLP models. CC is Conceptual Captions. B refers to BLEU-4, S to SPICE, M to METEOR, R to ROUGE-L, C to CIDEr and CH to CHAIR [115]. VG refers to Visual Genome and OI to OpenImages. Section 2 has more details on the datasets and metrics themselves. leaderboard. SCST is used for fine-tuning after initial training using cross-entropy optimization. For the weight updates performed during back propagation, the approach that was predominantly used was the Adam optimizer [117]. However, Conceptual Captions [59] uses Adagrad. Table 7 shows the details of the models that employ VLP. Compared to the non-VLP counterparts, these models tend to be much larger, especially when considering the number of layers and heads. The VLP models usually have several sizes for experimentation, which are typically labelled 'Base', 'Large' and 'Huge'. In Table 7, the best-performing size of each model is the one that has been taken into account. It is worth noting that many of the VLP models deviate from the use of Faster-RCNN as a feature extractor and instead use extractors based on vision transformers [99]. Faster-RCNN has served as a powerful feature extractor for a number of years but the effectiveness of transformer-based feature extractors has resulted in a shift in this area. This has yielded significant improvements in performance and has reduced the reliance on an explicit object detector. It has further shown that the vision part of vision-language models really does matter [39]. As seen from Table 7, the loss functions of the VLP methods are more varied. Apart from the objective loss function used during fine-tuning, they also have an objective loss function for the pre-training. For the pre-training, a common approach is to use the Masked Token Loss (MTL) [38]. The MTL is similar to the Masked Language Modeling (MLM) used in BERT [95]. Whereas in BERT the tokens only pertain to the language domain, in VLP the tokens can belong to the language or visual (image) domain. In Table 7, MTL-b refers to a Masked Token Loss that is bidirectional, whereas MTL-s is sequence-to-sequence [120]. CL2 and CL3 are two-way and three-way [38] contrastive loss functions, respectively.